使用mitmproxy + python做拦截代理

本文是一个较为完整的 mitmproxy 教程,侧重于介绍如何开发拦截脚本,帮助读者能够快速得到一个自定义的代理工具。

本文假设读者有基本的 python 知识,且已经安装好了一个 python 3 开发环境。如果你对 nodejs 的熟悉程度大于对 python,可移步到 anyproxy,anyproxy 的功能与 mitmproxy 基本一致,但使用 js 编写定制脚本。除此之外我就不知道有什么其他类似的工具了,如果你知道,欢迎评论告诉我。

本文基于 mitmproxy v4,当前版本号为 v4.0.1

继续阅读使用mitmproxy + python做拦截代理

读取的XML节点中带有冒号怎么办?

昨天,编程读取XML的时候,遇上了类似下面的一段XML

<a:root xmlns:a="http://ww.abc.com/">
    <a:book>aaaa</a:book>
</a:root>

起初没有特别的留意,于是乎就像平时读取XML一样使用了。

 var ele = from item in xDoc.Descendants("a:book") select item;

但是,运行报错,不允许传入冒号:之类的字符,后来查阅资料发现,节点中,冒号前的a代表是的命名空间,冒号后的才是根节点名称。在Root节点中,也对命名空间进行了声明 xmlns:a="http://ww.abc.com/" ,知道了这么一回事后,再来看看如何去读取,正确的读取是:

 XDocument xDoc = XDocument.Load("a.xml");
 XNamespace n = @"http://www.abc.com";
 var ele = from item in xDoc.Descendants(n + "book")
           select item.Value;

从代码可以看出,我声明了一个XNamespace类型的变量,并且把XML文件中出现的命名空间 http://ww.abc.com/ 赋值给它,然后再读取节点的时候,与真正的节点名称book进行拼接就可以了!

XML中出现命名空间的原因是,当你需要使用多个XML一起工作时,由于两个文档都包含带有不同内容和定义的节点元素,就会发生命名冲突,加上命名空间使用可以避免发生冲突,这与C#编程中类的命名空间的用处差不多。

另外,如果需要了解更多操作XML的可以访问下面这篇文章,写得很详细:

http://www.cnblogs.com/nsky/archive/2013/03/05/2944725.html

参考链接


Python操作Excel

Python使用openpyxl-3.0.3复制Excel的一行内容并且插入到下一行,包含公式,下拉框,示例代码如下:

# coding=utf-8
# openpyxl-3.0.3
from openpyxl.reader.excel import load_workbook
from openpyxl.cell.cell import TYPE_FORMULA
from openpyxl.worksheet.worksheet import Worksheet
from openpyxl.utils import get_column_letter
from openpyxl.worksheet.cell_range import CellRange
import re
import copy

def insert_rows(self, row_idx, cnt, above=False, copy_style=True, copy_merged_columns=True, fill_formulae=True):
    """Inserts new (empty) rows into worksheet at specified row index.

    :param row_idx: Row index specifying where to insert new rows.
    :param cnt: Number of rows to insert.
    :param above: Set True to insert rows above specified row index.
    :param copy_style: Set True if new rows should copy style of immediately above row.
    :param fill_formulae: Set True if new rows should take on formula from immediately above row, filled with references new to rows.

    Usage:

    * insert_rows(2, 10, above=True, copy_style=False)

    """
    CELL_RE  = re.compile("(?P<col>\$?[A-Z]+)(?P<row>\$?\d+)")

    row_idx = row_idx - 1 if above else row_idx
    def replace(m):
        row = m.group('row')
        prefix = "$" if row.find("$") != -1 else ""
        row = int(row.replace("$", ""))
        row += cnt if row > row_idx else 0
        return m.group('col') + prefix + str(row)

    # First, we shift all cells down cnt rows...
    old_cells = set()
    old_fas = set()
    new_cells = dict()
    new_fas = dict()
    for c in self._cells.values():

        old_coor = c.coordinate

        # Shift all references to anything below row_idx
        if c.data_type == TYPE_FORMULA:
            c.value = CELL_RE.sub(
                replace,
                c.value
            )
            # Here, we need to properly update the formula references to reflect new row indices
            if old_coor in self.formula_attributes and 'ref' in self.formula_attributes[old_coor]:
                self.formula_attributes[old_coor]['ref'] = CELL_RE.sub(
                    replace,
                    self.formula_attributes[old_coor]['ref']
                )

        # Do the magic to set up our actual shift
        if c.row > row_idx:
            old_coor = c.coordinate 
            old_cells.add((c.row, c.column))
            c.row += cnt
            new_cells[(c.row, c.column)] = c
            if old_coor in self.formula_attributes:
                old_fas.add(old_coor)
                fa = copy.copy(self.formula_attributes[old_coor])
                new_fas[c.coordinate] = fa

    for coor in old_cells:
        del self._cells[coor]
    self._cells.update(new_cells)

    for fa in old_fas:
        del self.formula_attributes[fa]
    self.formula_attributes.update(new_fas)

    # Next, we need to shift all the Row Dimensions below our new rows down by cnt...
    # CHANGED: for row in range(len(self.row_dimensions) - 1 + cnt, row_idx + cnt, -1):
    for row in range(list(self.row_dimensions)[-1] + cnt, row_idx + cnt, -1):
        new_rd = copy.copy(self.row_dimensions[row - cnt])
        new_rd.index = row
        self.row_dimensions[row] = new_rd
        del self.row_dimensions[row - cnt]

    # Now, create our new rows, with all the pretty cells
    # CHANGED: row_idx += 1
    new_row_idx = row_idx + 1
    for row in range(new_row_idx, new_row_idx + cnt):
        # Create a Row Dimension for our new row
        new_rd = copy.copy(self.row_dimensions[row-1])
        new_rd.index = row
        self.row_dimensions[row] = new_rd

        # CHANGED: for col in range(1,self.max_column):
        for col in range(self.max_column):
            col = col + 1
            cell = self.cell(row=row, column=col)
            source = self.cell(row=row_idx, column=col)
            if copy_style:
                cell.number_format = copy.copy(source.number_format)
                cell.font = copy.copy(source.font)
                cell.alignment = copy.copy(source.alignment)
                cell.border = copy.copy(source.border)
                cell.fill = copy.copy(source.fill)
            if fill_formulae and TYPE_FORMULA == source.data_type :
                s_coor = source.coordinate
                if s_coor in self.formula_attributes and 'ref' not in self.formula_attributes[s_coor]:
                    fa = copy.copy(self.formula_attributes[s_coor])
                    self.formula_attributes[cell.coordinate] = fa
                #print("Copying formula from cell %s%d to %s%d"%(col,row-1,col,row))
                cell.value = re.sub(
                    "(\$?[A-Z]{1,3}\$?)%d" % (row_idx),
                    lambda m: m.group(1) + str(row),
                    source.value
                ) 
                cell.data_type = TYPE_FORMULA

    # Check for Merged Cell Ranges that need to be expanded to contain new cells
    for cr in self.merged_cells.ranges:
        min_col, min_row, max_col, max_row = cr.bounds
        if min_row <= row_idx and max_row > row_idx:
            if max_row + cnt >= CellRange.max_row.max:
                cr.expand(down = CellRange.max_row.max - max_row)
            else:
                cr.expand(down = cnt)
        elif min_row > row_idx:
            if max_row + cnt >= CellRange.max_row.max:
                cr.expand(down = CellRange.max_row.max - max_row)
            else:
                cr.expand(down = cnt)
            cr.shrink(top = cnt)

    # Merge columns of the new rows in the same way row above does
    if copy_merged_columns:
        bounds = [] 
        for cr in self.merged_cells.ranges:
            if cr.max_row == cr.min_row == row_idx:
                bounds.append((cr.min_col, cr.max_col))
        for (min_col, max_col) in bounds:
            for row in range(new_row_idx, new_row_idx + cnt):
                newCellRange = get_column_letter(min_col) + str(row) + ":" + get_column_letter(max_col) + str(row)
                self.merge_cells(newCellRange)

    # update dataValidation
    validations = self.data_validations.dataValidation
    for val in validations: 
        for cr in val.cells:
            min_col, min_row, max_col, max_row = cr.bounds
            if min_row <= row_idx and max_row >= row_idx:
                if max_row + cnt >= CellRange.max_row.max:
                    cr.expand(down = CellRange.max_row.max - max_row)
                else:
                    cr.expand(down = cnt)
            elif min_row > row_idx:
                if max_row + cnt >= CellRange.max_row.max:
                    cr.expand(down = CellRange.max_row.max - max_row)
                else:
                    cr.expand(down = cnt)
                cr.shrink(top = cnt)

    # update conditional_formatting tow steps
    # first get all conditional_formatting need to update
    cond_fmts = self.conditional_formatting
    upd_cfs = []
    for cf in cond_fmts: 
        for cr in cf.cells:
            min_col, min_row, max_col, max_row = cr.bounds
            if min_row <= row_idx and max_row >= row_idx:
                upd_cfs.append(cf)
                break
            elif min_row > row_idx:
                upd_cfs.append(cf)
                break

    # second update conditional_formatting
    for cf in upd_cfs:
        rules = cond_fmts[cf]
        del cond_fmts[cf.cells]
        for cr in cf.cells:
            min_col, min_row, max_col, max_row = cr.bounds
            if min_row <= row_idx and max_row >= row_idx:
                if max_row + cnt >= CellRange.max_row.max:
                    cr.expand(down = CellRange.max_row.max - max_row)
                else:
                    cr.expand(down = cnt)
            elif min_row > row_idx:
                if max_row + cnt >= CellRange.max_row.max:
                    cr.expand(down = CellRange.max_row.max - max_row)
                else:
                    cr.expand(down = cnt)
                cr.shrink(top = cnt)
        for r in rules:
            cond_fmts[cf] = r


Worksheet.insert_rows = insert_rows

if __name__ == "__main__":
    # 注意,keep_vba=True打开的Excel文档,保存之后,MS Office 无法打开
    wb = load_workbook(filename='example.xlsx', read_only=False, keep_vba=False, data_only=False, keep_links=True)
    ws = wb.active
    ws.insert_rows(6, 4, above=True, copy_style=True)
    wb.save('new_document.xlsx')
    wb.close()

参考链接


Visual Studio Code调试Python无法Step Into导入(imported)模块的代码

Visual Studio Code调试Python无法Step Into导入(imported)外部模块的代码,导致在跟踪调用流程的时候非常不方便。

这个现象是Visual Studio Code为了防止调试的时候过多的关注不相干代码,因此默认禁止跟踪外部模块代码,只跟踪我们自己工程内部的代码。

禁止这部分功能,只需要在工程的配置文件launch.json中增加"justMyCode": false即可。

如下:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [

        {
            "name": "Python: 当前文件",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": false
        }
    ]
}

参考链接


PyQt5的QWebEngineView使用模板

一.支持视频播放

关键代码

self.settings().setAttribute(QWebEngineSettings.PluginsEnabled, True)      #支持视频播放
二.支持页面关闭请求

关键代码

self.page().windowCloseRequested.connect(self.on_windowCloseRequested)     #页面关闭请求
三.支持页面下载请求

关键代码

self.page().profile().downloadRequested.connect(self.on_downloadRequested) #页面下载请求

【如下代码,完全复制,直接运行,即可使用】

import sys
import os
import datetime
from PyQt5.QtWidgets import *
from PyQt5.QtCore import *
from PyQt5.QtWebEngineWidgets import QWebEngineView,QWebEngineSettings

# 调试窗口配置
# 如果不想自己创建调试窗口,可以使用Chrome连接这个地址进行调试
DEBUG_PORT = '5588'
DEBUG_URL = 'http://127.0.0.1:%s' % DEBUG_PORT
os.environ['QTWEBENGINE_REMOTE_DEBUGGING'] = DEBUG_PORT

################################################
#######创建主窗口
################################################
class MainWindow(QMainWindow):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.setWindowTitle('My Browser')
        #self.showMaximized()
        self.setWindowFlags(Qt.FramelessWindowHint)

        #####创建tabwidget
        self.tabWidget = QTabWidget()
        self.tabWidget.setTabShape(QTabWidget.Triangular)
        self.tabWidget.setDocumentMode(True)
        self.tabWidget.setMovable(True)
        self.tabWidget.setTabsClosable(True)
        self.tabWidget.tabCloseRequested.connect(self.close_Tab)
        self.setCentralWidget(self.tabWidget)

        ####第一个tab
        self.webview = WebEngineView(self)   #self必须要有,是将主窗口作为参数,传给浏览器
        self.webview.load(QUrl("https://www.baidu.com"))
        self.create_tab(self.webview)

        #网页调试窗口
        self.inspector = QWebEngineView()
        self.inspector.setWindowTitle('Web Inspector')
        self.inspector.load(QUrl(DEBUG_URL))
        self.webview.loadFinished.connect(self.handleHtmlLoaded)        

    # 加载完成后显示调试网页
    def handleHtmlLoaded(self, ok):
        if ok:
            self.webview.page().setDevToolsPage(self.inspector.page())
            self.inspector.show()

    #创建tab
    def create_tab(self,webview):
        self.tab = QWidget()
        self.tabWidget.addTab(self.tab, "新标签页")
        self.tabWidget.setCurrentWidget(self.tab)
        #####
        self.Layout = QHBoxLayout(self.tab)
        self.Layout.setContentsMargins(0, 0, 0, 0)
        self.Layout.addWidget(webview)

    #关闭tab
    def close_Tab(self,index):
        if self.tabWidget.count()>1:
            self.tabWidget.removeTab(index)
        else:
            self.close()   # 当只有1个tab时,关闭主窗口

################################################
#######创建浏览器
################################################
class WebEngineView(QWebEngineView):

    def __init__(self,mainwindow,parent=None):
        super(WebEngineView, self).__init__(parent)
        self.mainwindow = mainwindow
        ##############
        self.settings().setAttribute(QWebEngineSettings.PluginsEnabled, True)      #支持视频播放
        self.page().windowCloseRequested.connect(self.on_windowCloseRequested)     #页面关闭请求
        self.page().profile().downloadRequested.connect(self.on_downloadRequested) #页面下载请求

    #  支持页面关闭请求
    def on_windowCloseRequested(self):
        the_index = self.mainwindow.tabWidget.currentIndex()
        self.mainwindow.tabWidget.removeTab(the_index)


    #  支持页面下载按钮
    def on_downloadRequested(self,downloadItem):
        if  downloadItem.isFinished()==False and downloadItem.state()==0:
            ###生成文件存储地址
            the_filename = downloadItem.url().fileName()
            if len(the_filename) == 0 or "." not in the_filename:
                cur_time = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
                the_filename = "下载文件" + cur_time + ".xls"
            the_sourceFile = os.path.join(os.getcwd(), the_filename)

            ###下载文件
            # downloadItem.setSavePageFormat(QWebEngineDownloadItem.CompleteHtmlSaveFormat)
            downloadItem.setPath(the_sourceFile)
            downloadItem.accept()
            downloadItem.finished.connect(self.on_downloadfinished)


    #  下载结束触发函数
    def on_downloadfinished(self):
        js_string = '''
        alert("下载成功,请到软件同目录下,查找下载文件!"); 
        '''
        self.page().runJavaScript(js_string)


    # 重写createwindow()
    def createWindow(self, QWebEnginePage_WebWindowType):
        new_webview = WebEngineView(self.mainwindow)

        self.mainwindow.create_tab(new_webview)

        return new_webview


################################################
#######程序入门
################################################
if __name__ == "__main__":
    app = QApplication(sys.argv)
    the_mainwindow = MainWindow()
    the_mainwindow.show()
    sys.exit(app.exec())

参考链接


PyQt WebEngineView interferes with MainMenu

I'm trying to create an application that contains a web browser within it, but when I add the web browser my menu bar visually disappears but functionally remains in place. The following are two images, one showing the "self.centralWidget(self.web_widget)" commented out, and the other allows that line to run. If you run the example code, you will also see that while visually the entire web page appears as if the menu bar wasn't present, you have to click slightly below each entry field and button in order to activate it, behaving as if the menu bar was in fact present.

继续阅读PyQt WebEngineView interferes with MainMenu

macOS Catalina(10.15.4)下PyQt5图标Icon无法显示的问题

macOS下,刚学习PyQt5遇到图标无法显示问题. 先上代码,看下代码就明白了。

import sys
from PyQt5.QtWidgets import QApplication, QWidget
from PyQt5.QtGui import QIcon, QPixmap
from PyQt5.QtCore import QFile
 
 
# 面向对象编程
class main_widget(QWidget):  # 继承自 QWidget类
    def __init__(self):
        super().__init__()
        self.initUI()  # 创建窗口
 
 
    def initUI(self):  #  在此处添加 窗口控件
        self.setGeometry(300, 200, 600, 500)  # 屏幕上坐标(x, y), 和 窗口大小(宽,高)
        self.setWindowTitle("第一个qt5窗口")
        # self.setWindowIcon(QIcon("logo/m4.png"))  # 设置窗口图标
        self.show()
 
 
if __name__ == "__main__":
    app = QApplication(sys.argv)
    path = 'logo/m4.png'
    print(QFile.exists(path))
    app.setWindowIcon(QIcon(path))  # MAC 下 程序图标是显示在程序坞中的, 切记;
    window = main_widget()
    sys.exit(app.exec())  # 退出主循环

setWindowIconQApplication的方法,而不是QWidget的,所以使用app.setWindowIcon设置是对的。

注意在macOS下,图标是显示在程序坞中的!!!

参考链接


MAC下PyQt5图标Icon无法显示的问题