UTF8 + BOM产生问题与小结

写python脚本的时候发现这样一个问题：从xls文件导出到txt时，无法直接转换为int型数据，输出查看发现和文件编码方式产生的附加信息有关用一个简单的文件举例

90905

90907

90908

90909

90939

90940

90946

90959

90961

90965

当文件分别用ascii，utf8，utf8+bom作为编码格式时，显示输出结果如下：

使用ascii编码的输出：

['90905\r\n', '90907\r\n', '90908\r\n', '90909\r\n', '90939\r\n', '90940\r\n', '90946\r\n', '90959\r\n', '90961\r\n', '90965']

1	['90905\r\n', '90907\r\n', '90908\r\n', '90909\r\n', '90939\r\n', '90940\r\n', '90946\r\n', '90959\r\n', '90961\r\n', '90965']

使用utf8编码的输出：

['90905\r\n', '90907\r\n', '90908\r\n', '90909\r\n', '90939\r\n', '90940\r\n', '90946\r\n', '90959\r\n', '90961\r\n', '90965']

1	['90905\r\n', '90907\r\n', '90908\r\n', '90909\r\n', '90939\r\n', '90940\r\n', '90946\r\n', '90959\r\n', '90961\r\n', '90965']

使用bom编码的输出：

['\xef\xbb\xbf90905\r\n', '90907\r\n', '90908\r\n', '90909\r\n', '90939\r\n', '90940\r\n', '90946\r\n', '90959\r\n', '90961\r\n', '90965']

1	['\xef\xbb\xbf90905\r\n', '90907\r\n', '90908\r\n', '90909\r\n', '90939\r\n', '90940\r\n', '90946\r\n', '90959\r\n', '90961\r\n', '90965']

原来utf8+bom不能直接转换int的原因在这里，它在文件头插入了一个表示文件编码的信息\xef\xbb\xbf，那么UTF-8(无BOM）和UTF－8这两个有什么区别呢？BOM是什么呢？

什么是BOM？

BOM: Byte Order Mark

UTF-8 BOM又叫UTF-8 签名,其实UTF-8 的BOM对UFT-8没有作用,是为了支持UTF-16,UTF-32才加上的

BOM,BOM签名的意思就是告诉编辑器当前文件采用何种编码,方便编辑器识别,但是BOM虽然在编辑器中不显示,但是会产生输出,就像多了一个空行。

Byte Order Marks are special characters at the beginning of a Unicode file to indicate whether it is big or little endian, in other words does the high or low order byte come first. These codes also tell whether the encoding is 8, 16 or 32 bit. You can recognise Unicode files by their starting byte order marks, and by the way Unicode-16 files are half zeroes and Unicode-32 files are three-quarters zeros. Unicode Endian Markers

Byte-order mark Description
EF BB BF UTF-8
FF FE UTF-16 aka UCS-2, little endian
FE FF UTF-16 aka UCS-2, big endian
00 00 FF FE UTF-32 aka UCS-4, little endian.
00 00 FE FF UTF-32 aka UCS-4, big-endian.

UTF的字节序和BOM

UTF- 8以字节为编码单元，没有字节序的问题。UTF-16以两个字节为编码单元，在解释一个UTF-16文本前，首先要弄清楚每个编码单元的字节序。例如收到一个“奎”的Unicode编码是594E，“乙”的Unicode编码是4E59。如果我们收到UTF-16字节流“594E”，那么这是“奎”还是 “乙”？

Unicode规范中推荐的标记字节顺序的方法是BOM。BOM不是“Bill Of Material”的BOM表，而是Byte Order Mark。BOM是一个有点小聪明的想法：

在 UCS编码中有一个叫做"ZERO WIDTH NO-BREAK SPACE"的字符，它的编码是FEFF。而FFFE在UCS中是不存在的字符，所以不应该出现在实际传输中。UCS规范建议我们在传输字节流前，先传输字符"ZERO WIDTH NO-BREAK SPACE"。

这样如果接收者收到FEFF，就表明这个字节流是Big-Endian的；如果收到FFFE，就表明这个字节流是Little-Endian的。因此字符"ZERO WIDTH NO-BREAK SPACE"又被称作BOM。

UTF-8不需要BOM来表明字节顺序，但可以用BOM来表明编码方式。字符"ZERO WIDTH NO-BREAK SPACE"的UTF-8编码是EF BB BF。所以如果接收者收到以EF BB BF开头的字节流，就知道这是UTF-8编码了。

Windows就是使用BOM来标记文本文件的编码方式的。

原来BOM是在文件的开始加了几个字节作为标记。有了这个标记，一些协议和系统才能识别。

ok,说了这么多背景，那么如何解决这个问题呢？

如何使用BOM头

BOM头的删除

对UTF-16, Python将BOM解码为空字串。然而对UTF-8, BOM被解码为一个字符，如例：

>>> codecs.BOM_UTF16.decode( "utf16" )
>>> codecs.BOM_UTF8.decode( "utf8" )
 u'\ufeff'

>>> codecs.BOM_UTF16.decode( "utf16" )

>>> codecs.BOM_UTF8.decode( "utf8" )

u'\ufeff'

简单的做法是在文件读入时使用

import codecs

f = codecs.open(sys.argv[1],'r', 'utf_8_sig')

import codecs

f = codecs.open(sys.argv[1],'r', 'utf_8_sig')

即可，具体可以参见[http://docs.python.org/library/codecs.html#module-encodings.utf_8_sig|http://docs.python.org/library/codecs.html#module-encodings.utf_8_sig]

或者：

u.lstrip( unicode( codecs.BOM_UTF8, "utf8" ) )

1	u.lstrip( unicode( codecs.BOM_UTF8, "utf8" ) )

BOM头的添加

out = file( "someFile", "w" )
out.write( codecs.BOM_UTF8 )
out.write( unicodeString.encode( "utf-8" ) )
out.close()
out = file( "someFile", "w" )
out.write( codecs.BOM_UTF8 )
out.write( unicodeString.encode( "utf-8" ) )
out.close()

out = file( "someFile", "w" )

out.write( codecs.BOM_UTF8 )

out.write( unicodeString.encode( "utf-8" ) )

out.close()

out = file( "someFile", "w" )

out.write( codecs.BOM_UTF8 )

out.write( unicodeString.encode( "utf-8" ) )

out.close()

参考 http://www.cnblogs.com/DDark/archive/2011/11/28/2266085.html

python下的编码检测——chardet

在处理字符串时，常常会遇到不知道字符串是何种编码，如果不知道字符串的编码就不能将字符串转换成需要的编码。面对多种不同编码的输入方式，是否会有一种有效的编码方式？chardet是一个非常优秀的编码识别模块。

chardet 是python的第三方库，需要下载和安装。现在 pip 已经可以很好的支持这个版本的下载了，建议使用pip 安装，关于pip 的安装部分，可以参考Windows下python的包管理器pip安装

在安装完chardet模块，我就可以使用它了，来看一段示例代码。

import chardet
import urllib

#可根据需要，选择不同的数据
TestData = urllib.urlopen('http://www.baidu.com/').read()
print chardet.detect(TestData)

运行结果：
{'confidence': 0.99, 'encoding': 'GB2312'}

import chardet

import urllib

#可根据需要，选择不同的数据

TestData = urllib.urlopen('http://www.baidu.com/').read()

print chardet.detect(TestData)

运行结果：

{'confidence': 0.99, 'encoding': 'GB2312'}

运行结果表示有99%的概率认为这段代码是GB2312编码方式。

另外一个相对高级的应用。

import urllib
from chardet.universaldetector import UniversalDetector
usock = urllib.urlopen('http://www.baidu.com/')
#创建一个检测对象
detector = UniversalDetector()
for line in usock.readlines():
	#分块进行测试，直到达到阈值
    detector.feed(line)
    if detector.done: break
#关闭检测对象
detector.close()
usock.close()
#输出检测结果
print detector.result

运行结果：
{'confidence': 0.99, 'encoding': 'GB2312'}

import urllib

from chardet.universaldetector import UniversalDetector

usock = urllib.urlopen('http://www.baidu.com/')

#创建一个检测对象

detector = UniversalDetector()

for line in usock.readlines():

#分块进行测试，直到达到阈值

detector.feed(line)

if detector.done: break

#关闭检测对象

detector.close()

usock.close()

#输出检测结果

print detector.result

运行结果：

{'confidence': 0.99, 'encoding': 'GB2312'}

应用背景，如果要对一个大文件进行编码识别，使用这种高级的方法，可以只读一部，去判别编码方式从而提高检测速度。

参考 http://blog.csdn.net/aqwd2008/article/details/7506007

Windows下python的包管理器pip安装

做python开发，要用到第三方包，关于MAC下面如何安装Pip ，可以参考前面的 Mac OS 10.9 下python安装easy_install pip 现在我们看看在Windows下面的操作，打开 Pip的官方安装指南https://pip.pypa.io/en/latest/installing.html

Installation

Python & OS Support

pip works with CPython versions 2.6, 2.7, 3.1, 3.2, 3.3, 3.4 and also pypy.

pip works on Unix/Linux, OS X, and Windows.

Note

Python 2.5 was supported through v1.3.1, and Python 2.4 was supported through v1.1.

Install pip

To install or upgrade pip, securely download get-pip.py.（如果官网下载不下来，可以点这个链接在本网站下载）

Then run the following (which may require administrator access):

python get-pip.py

				1

						python get-pip.py

If setuptools (or distribute) is not already installed, get-pip.py will install setuptools for you. [2]

To upgrade an existing setuptools (or distribute), run pip install -U setuptools. [3]

To enable the use of pip from the command line, ensure the Scripts subdirectory of your Python installation is available on the system PATH. (This is not done automatically.)

Additionally, get-pip.py supports using the pip install options and the general options. Below are some examples:

Install from local copies of pip and setuptools:

python get-pip.py --no-index --find-links=/local/copies

				1

						python get-pip.py --no-index --find-links=/local/copies

Install to the user site [4]:

python get-pip.py --user

				1

						python get-pip.py --user

Install behind a proxy:

python get-pip.py --proxy="[user:passwd@]proxy.server:port"

				1

						python get-pip.py --proxy="[user:passwd@]proxy.server:port"

Upgrade pip

On Linux or OS X:

pip install -U pip

				1

						pip install -U pip

On Windows [5]:

python -m pip install -U pip

				1

						python -m pip install -U pip

Using Package Managers

On Linux, pip will generally be available for the system install of python using the system package manager, although often the latest version will be unavailable.

On Debian and Ubuntu:

sudo apt-get install python-pip

				1

						sudo apt-get install python-pip

On Fedora:

sudo yum install python-pip

				1

						sudo yum install python-pip

Mac OS 10.9/macOS High Sierra下python安装pip

pip是一个安装python库很方便的东西，类似yum，pip search pip install.

安装 pip

#https://github.com/pypa/get-pip
$ curl https://bootstrap.pypa.io/get-pip.py | sudo python

1 2	#https://github.com/pypa/get-pip $ curl https://bootstrap.pypa.io/get-pip.py \| sudo python

python在windows的cmd中打印彩色文字

#!/usr/bin/env python
#encoding: utf-8
import ctypes

STD_INPUT_HANDLE = -10
STD_OUTPUT_HANDLE= -11
STD_ERROR_HANDLE = -12

FOREGROUND_BLACK = 0x0
FOREGROUND_BLUE = 0x01 # text color contains blue.
FOREGROUND_GREEN= 0x02 # text color contains green.
FOREGROUND_RED = 0x04 # text color contains red.
FOREGROUND_INTENSITY = 0x08 # text color is intensified.

BACKGROUND_BLUE = 0x10 # background color contains blue.
BACKGROUND_GREEN= 0x20 # background color contains green.
BACKGROUND_RED = 0x40 # background color contains red.
BACKGROUND_INTENSITY = 0x80 # background color is intensified.

class Color:
 ''' See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winprog/winprog/windows_api_reference.asp
 for information on Windows APIs.'''
 std_out_handle = ctypes.windll.kernel32.GetStdHandle(STD_OUTPUT_HANDLE)

 def set_cmd_color(self, color, handle=std_out_handle):
   """(color) -> bit
Example: set_cmd_color(FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_BLUE | FOREGROUND_INTENSITY)
"""
  bool = ctypes.windll.kernel32.SetConsoleTextAttribute(handle, color)
  return bool

def reset_color(self):
  self.set_cmd_color(FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_BLUE)

def print_red_text(self, print_text):
  self.set_cmd_color(FOREGROUND_RED | FOREGROUND_INTENSITY)
  print print_text
  self.reset_color()

def print_green_text(self, print_text):
  self.set_cmd_color(FOREGROUND_GREEN | FOREGROUND_INTENSITY)
  print print_text
  self.reset_color()

def print_blue_text(self, print_text):
  self.set_cmd_color(FOREGROUND_BLUE | FOREGROUND_INTENSITY)
  print print_text
  self.reset_color()

def print_red_text_with_blue_bg(self, print_text):
  self.set_cmd_color(FOREGROUND_RED | FOREGROUND_INTENSITY| BACKGROUND_BLUE | BACKGROUND_INTENSITY)
  print print_text
  self.reset_color()

if __name__ == "__main__":
 clr = Color()
 clr.print_red_text('red')
 clr.print_green_text('green')
 clr.print_blue_text('blue')
 clr.print_red_text_with_blue_bg('background')

#!/usr/bin/env python

#encoding: utf-8

import ctypes

STD_INPUT_HANDLE = -10

STD_OUTPUT_HANDLE= -11

STD_ERROR_HANDLE = -12

FOREGROUND_BLACK = 0x0

FOREGROUND_BLUE = 0x01 # text color contains blue.

FOREGROUND_GREEN= 0x02 # text color contains green.

FOREGROUND_RED = 0x04 # text color contains red.

FOREGROUND_INTENSITY = 0x08 # text color is intensified.

BACKGROUND_BLUE = 0x10 # background color contains blue.

BACKGROUND_GREEN= 0x20 # background color contains green.

BACKGROUND_RED = 0x40 # background color contains red.

BACKGROUND_INTENSITY = 0x80 # background color is intensified.

class Color:

''' See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winprog/winprog/windows_api_reference.asp

for information on Windows APIs.'''

std_out_handle = ctypes.windll.kernel32.GetStdHandle(STD_OUTPUT_HANDLE)

def set_cmd_color(self, color, handle=std_out_handle):

"""(color) -> bit

Example: set_cmd_color(FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_BLUE | FOREGROUND_INTENSITY)

"""

bool = ctypes.windll.kernel32.SetConsoleTextAttribute(handle, color)

return bool

def reset_color(self):

self.set_cmd_color(FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_BLUE)

def print_red_text(self, print_text):

self.set_cmd_color(FOREGROUND_RED | FOREGROUND_INTENSITY)

print print_text

self.reset_color()

def print_green_text(self, print_text):

self.set_cmd_color(FOREGROUND_GREEN | FOREGROUND_INTENSITY)

print print_text

self.reset_color()

def print_blue_text(self, print_text):

self.set_cmd_color(FOREGROUND_BLUE | FOREGROUND_INTENSITY)

print print_text

self.reset_color()

def print_red_text_with_blue_bg(self, print_text):

self.set_cmd_color(FOREGROUND_RED | FOREGROUND_INTENSITY| BACKGROUND_BLUE | BACKGROUND_INTENSITY)

print print_text

self.reset_color()

if __name__ == "__main__":

clr = Color()

clr.print_red_text('red')

clr.print_green_text('green')

clr.print_blue_text('blue')

clr.print_red_text_with_blue_bg('background')

一	二	三	四	五	六	日
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31

分类： Python

Python实现抓取CSDN首页文章列表

UTF8 + BOM产生问题与小结

什么是BOM？

BOM: Byte Order Mark

UTF的字节序和BOM

如何使用BOM头

BOM头的删除

BOM头的添加

python下的编码检测——chardet

Windows下python的包管理器pip安装

Installation

Python & OS Support

Install pip

Upgrade pip

Using Package Managers

Mac OS 10.9/macOS High Sierra下python安装pip

python在windows的cmd中打印彩色文字