error: call to ‘__open_missing_mode’ declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments

使用open函数的时候,如果在第二个参数中使用了O_CREAT,就必须添加第三个参数:创建文件时赋予的初始权。

In function ‘open’,
    inlined from ‘test_detector_ffmpeg’ at ./src/ff_darknet.c:421:15:
/usr/include/x86_64-linux-gnu/bits/fcntl2.h:50:4: error: call to ‘__open_missing_mode’ declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments
    __open_missing_mode ();
    ^
compilation terminated due to -Wfatal-errors.

建议使用如下参数创建文件:

//0755
fd = open(file_path,O_CREAT | O_APPEND | O_WRONLY,S_IRWXU|S_IRUSR|S_IXUSR|S_IROTH|S_IXOTH);

Ubuntu 16.04/14.04.5安装Nvidia CUDA驱动

Ubuntu 16.04/14.04.5上已经可以简化到直接用命令行来安装Nvidia CUDA驱动了,不需要以往的繁琐操作,只是安装的版本比较老,但是目前已经足够使用了。

$ sudo apt install nvidia-cuda-toolkit

安装的版本目前是Nvidia CUDA 7.5(Ubuntu 16.04)/Nvidia CUDA 5.5(Ubuntu 14.04.5)版本,最新的Nvidia CUDA 8.0版本还是需要从Nvidia官网下载,然后手工安装才行。

A Practical Introduction to Deep Learning with Caffe and Python

Deep learning is the new big trend in machine learning. It had many recent successes in computer vision, automatic speech recognition and natural language processing.

The goal of this blog post is to give you a hands-on introduction to deep learning. To do this, we will build a Cat/Dog image classifier using a deep learning algorithm called convolutional neural network (CNN) and a Kaggle dataset.

This post is divided into 2 main parts. The first part covers some core concepts behind deep learning, while the second part is structured in a hands-on tutorial format.

In the first part of the hands-on tutorial (section 4), we will build a Cat/Dog image classifier using a convolutional neural network from scratch. In the second part of the tutorial (section 5), we will cover an advanced technique for training convolutional neural networks called transfer learning. We will use some Python code and a popular open source deep learning framework called Caffe to build the classifier. Our classifier will be able to achieve a classification accuracy of 97%.

By the end of this post, you will understand how convolutional neural networks work, and you will get familiar with the steps and the code for building these networks.

The source code for this tutorial can be found in this github repository.

继续阅读A Practical Introduction to Deep Learning with Caffe and Python

Deep learning tutorial on Caffe technology : basic commands, Python and C++ code.

Caffe is certainly one of the best frameworks for deep learning, if not the best.

Let’s try to put things into order, in order to get a good tutorial :).

Caffe

Install

First install Caffe following my tutorials on Ubuntu or Mac OS with Python layers activated and pycaffe path correctly set export PYTHONPATH=~/technologies/caffe/python/:$PYTHONPATH.

继续阅读Deep learning tutorial on Caffe technology : basic commands, Python and C++ code.

macOS Sierra (10.12.4)下Caffe执行Python代码报告错误“Mean shape incompatible with input shape”

在执行macOS Sierra (10.12.4)下Caffe通过Python接口加载binaryproto格式的均值文件的时候,最后报告错误:

Traceback (most recent call last):
  File "analysis_memnet.py", line 29, in <module>
    detector = caffe.Detector(model_def, pretrained_model, mean=means)
  File "/Users/Source/caffe/distribute/python/caffe/detector.py", line 46, in __init__
    self.transformer.set_mean(in_, mean)
  File "/Users/Source/caffe/distribute/python/caffe/io.py", line 259, in set_mean
    raise ValueError('Mean shape incompatible with input shape.')
ValueError: Mean shape incompatible with input shape.

这个错误发生的原因是由于memnet提供的均值文件是256*256的,但是提供的配置文件却是227*227的,导致在io.py里面的代码在进行判断的时候发生异常。调整源代码中的python/caffe/io.py里面的代码:

    def set_mean(self, in_, mean):
        """
        Set the mean to subtract for centering the data.

        Parameters
        ----------
        in_ : which input to assign this mean.
        mean : mean ndarray (input dimensional or broadcastable)
        """
        self.__check_input(in_)
        ms = mean.shape
        if mean.ndim == 1:
            # broadcast channels
            if ms[0] != self.inputs[in_][1]:
                raise ValueError('Mean channels incompatible with input.')
            mean = mean[:, np.newaxis, np.newaxis]
        else:
            # elementwise mean
            if len(ms) == 2:
                ms = (1,) + ms
            if len(ms) != 3:
                raise ValueError('Mean shape invalid')
            if ms != self.inputs[in_][1:]:
                raise ValueError('Mean shape incompatible with input shape.')
        self.mean[in_] = mean

调整为:

    def set_mean(self, in_, mean):
        """
        Set the mean to subtract for centering the data.

        Parameters
        ----------
        in_ : which input to assign this mean.
        mean : mean ndarray (input dimensional or broadcastable)
        """
        self.__check_input(in_)
        ms = mean.shape
        if mean.ndim == 1:
            # broadcast channels
            if ms[0] != self.inputs[in_][1]:
                raise ValueError('Mean channels incompatible with input.')
            mean = mean[:, np.newaxis, np.newaxis]
        else:
            # elementwise mean
            if len(ms) == 2:
                ms = (1,) + ms
            if len(ms) != 3:
                raise ValueError('Mean shape invalid')
            if ms != self.inputs[in_][1:]:
                in_shape = self.inputs[in_][1:]
                m_min, m_max = mean.min(), mean.max()
                normal_mean = (mean - m_min) / (m_max - m_min)
                mean = resize_image(normal_mean.transpose((1,2,0)),in_shape[1:]).transpose((2,0,1)) * (m_max - m_min) + m_min
                #raise ValueError('Mean shape incompatible with input shape.')
        self.mean[in_] = mean

调整完成后,需要重新编译Caffe:

$ make clean
$ make
$ make pycaffe
$ make distribute

参考链接


macOS Sierra (10.12.4)下Caffe通过Python接口加载binaryproto格式的均值文件

macOS Sierra (10.12.4)下Caffe通过Python接口加载均值文件的时候,都是加载的.npy格式的文件,这个格式是Python存储的格式,跟我们经常下载到的.binaryproto格式的均值文件是不同的,这样就导致了加载问题。
.binaryprotoGoogleProtocol Buffer序列化后的数据,而.npy格式是Pythonnumpy模块序列化后的数据。

之所以会出现两种不同的存储格式,目前猜测是由于目前Python 3不能很好的支持Protocol Buffer导致的。

Python下是不能直接加载.binaryproto格式的数据的,必须进行一次转换才行,示例代码如下:

#coding=utf-8
#加载必要的库
import numpy as np
 
import sys,os

#设置当前目录
caffe_root = '/Users/Source/caffe/distribute/'
sys.path.append(caffe_root + 'python')

import caffe
os.chdir(caffe_root)

memnet_root = '/Users/Source/caffe/'
model_def =memnet_root + 'models/memnet/deploy.prototxt'
pretrained_model=memnet_root + 'models/memnet/memnet.caffemodel'
means_file=memnet_root + 'models/memnet/mean.binaryproto'

caffe.set_mode_cpu() 
blobProto=caffe.io.caffe_pb2.BlobProto()
binProtoFile=open(means_file,'rb')
blobProto.ParseFromString(binProtoFile.read())
means = caffe.io.blobproto_to_array(blobProto)[0]
binProtoFile.close()

# Make detector.
detector = caffe.Detector(model_def, pretrained_model, mean=means)

参考链接


macOS Sierra (10.12.4)编译pycaffe成功后,执行时候崩溃,错误“Segmentation fault: 11”

参照 macOS Sierra (10.12.3)编译Caffe 编译成功 Caffe 后,开始尝试使用 CaffePython 接口,执行如下命令:

$ make pycaffe

编译一切成功,但是当执行

import caffe

的时候,程序崩溃,提示如下内容:

Segmentation fault: 11

继续阅读macOS Sierra (10.12.4)编译pycaffe成功后,执行时候崩溃,错误“Segmentation fault: 11”

macOS Sierra (10.12.4)下Python通过PyAV调用FFMPEG操作视频

macOS Sierra (10.12.4)下使用Python操作视频,FFMPEG是目前来说最好的一个选择,但是没有为Python专门提供适配接口,网上搜索了比较长时间,才找到PyAV来操作FFMPEG

PyAV的文档地址在:https://mikeboers.github.io/PyAV/

代码地址在:https://github.com/mikeboers/PyAV

首先需要通过HomeBrew安装FFMPEG

$ brew install ffmpeg

接下来安装PyAV,安装方式两种:

一种是直接通过PIP来安装:

$ pip install av

另外一种是通过下载代码来手工安装

$ git clone https://github.com/mikeboers/PyAV.git
$ cd PyAV
$ python setup.py install

安装好后的例子如下:

import av
from av.frame import Frame
from av.packet import Packet
from av.stream import Stream
from av.utils import AVError
from av.video import VideoFrame

container = av.open('san.mp4')

for frame in container.decode(video=0):
    frame.to_image().save('./pyav/frame-%04d.jpg' % frame.index)