Файловый менеджер - Редактировать - /opt/imh-python/lib/python3.9/site-packages/parso/python/__pycache__/diff.cpython-39.pyc
Ðазад
a e�h�� � @ sX d Z ddlZddlZddlmZ ddlZddlmZ ddlm Z ddl m Z ddl m Z mZ ddlmZ e�e�Zd Zd ZejZejZejZejZejZd d � Zd d� Z dd� Z dd� Z dd� Z dd� Z dd� Z!dd� Z"dd � Z#d d � Z$d4d d!�Z%d"d#� Z&d$d%� Z'd&d'� Z(d(d)� Z)G d*d+� d+e*�Z+d,d-� Z,G d.d/� d/�Z-G d0d1� d1�Z.G d2d3� d3�Z/dS )5aa The diff parser is trying to be a faster version of the normal parser by trying to reuse the nodes of a previous pass over the same file. This is also called incremental parsing in parser literature. The difference is mostly that with incremental parsing you get a range that needs to be reparsed. Here we calculate that range ourselves by using difflib. After that it's essentially incremental parsing. The biggest issue of this approach is that we reuse nodes in a mutable way. The intial design and idea is quite problematic for this parser, but it is also pretty fast. Measurements showed that just copying nodes in Python is simply quite a bit slower (especially for big files >3 kLOC). Therefore we did not want to get rid of the mutable nodes, since this is usually not an issue. This is by far the hardest software I ever wrote, exactly because the initial design is crappy. When you have to account for a lot of mutable state, it creates a ton of issues that you would otherwise not have. This file took probably 3-6 months to write, which is insane for a parser. There is a fuzzer in that helps test this whole thing. Please use it if you make changes here. If you run the fuzzer like:: test/fuzz_diff_parser.py random -n 100000 you can be pretty sure that everything is still fine. I sometimes run the fuzzer up to 24h to make sure everything is still ok. � N)� namedtuple)� split_lines)�Parser)� EndMarker)� PythonToken�BOM_UTF8_STRING)�PythonTokenTypesF)�INDENT� ERROR_DEDENT�DEDENTc C s | j dko| jtv S )N� error_leaf)�type� token_type�_INDENTATION_TOKENS��node� r ��/root/rpmbuild/BUILDROOT/imh-python39-modules-3.9.7-92.el8.x86_64/opt/imh-python/lib/python3.9/site-packages/parso/python/diff.py�_is_indentation_error_leaf3 s r c C s | rt | �r| �� } q | S �N)r �get_previous_leaf��leafr r r �!_get_previous_leaf_if_indentation7 s r c C s | rt | �r| �� } q | S r )r � get_next_leafr r r r � _get_next_leaf_if_indentation= s r c C s t | jd �S �N� )�_get_indentation�children�� tree_noder r r �_get_suite_indentationC s r" c C s | j d S r )� start_posr r r r r G s r c C sh z | j }W �n* t�y6 | jdkrJ| jtv rJ| jr:J �| jrDJ �Y dS t| �� �}|du rj| j}d}n*|j | j ks�J || f��|j| j }|j }d|v s�d|v r�t |�}|d t |� d }|t |d �f}nF|d |d t |� f}|� t��r| �� dk�r|d |d d f}| j |k�s2J | j |f��Y n.0 |D ]&}|j| k�sXJ | |f��t|� �q<dS ) z~ Checks if the parent/children relationship is correct. This is a check that only runs during debugging/testing. r N)r r � � r r ���)r �AttributeErrorr r r �value�prefixr r �end_posr# r �len� startswithr �get_start_pos_of_prefix�parent�_assert_valid_graph) r r Z previous_leaf�contentZprevious_start_posZsplitted�line�actual�childr r r r/ K s: � � r/ c C s z | j }W n� ty� t|d�r.J | |f��| j|jksFJ | |f��| j|jks^J | |f��| j|jksvJ | |f��| j|jks�J | |f��Y d S 0 z |j }W n" ty� ds�J | |f��Y n0 t||�D ]\}}t||� q�t |�t |�k�sJ dt |� d t |� ��d S )Nr Fr$ ) r r' �hasattrr( r r) r# �zip�_assert_nodes_are_equalr+ �repr)Znode1Znode2Z children1Z children2Zn1Zn2r r r r6 y s r6 c C sL t | �� dd�}t�||�}t�||�}dd l}d|jd�|�d�|�f S )NT��keependsr zmThere's an issue with the diff parser. Please report (parso v%s) - Old/New: %s Actual Diff (May be empty): %s� )r �get_code�difflibZ unified_diff�parso� __version__�join)�module� old_lines� new_linesZ current_linesZ current_diffZ old_new_diffr= r r r �_get_debug_error_message� s ��rC c C sR | � � }t|�r|jd S |�� }|jdkrDd|jv rD|jd d S |jd S d S )Nr � endmarkerr$ r )� get_last_leaf�_ends_with_newliner# r r r) r* )Z node_or_leaf� last_leaf�nr r r �_get_last_line� s rI c C s* | d ur&| j dkr&| jdkr&| �� } q | S )Nr r )r r r r r r r �_skip_dedent_error_leaves� s rJ r: c C s@ t | �} | jdkr | j�� }n| j}|dkp>|�d�p>|�d�S )Nr �newliner$ r% )rJ r r �lower�endswith)r �suffix�typr r r rF � s rF c C s |D ]}|j dv r dS qdS )zg if, while, for and try might not be finished, because another part might still be parsed. )Zif_stmtZ while_stmtZfor_stmtZtry_stmtFT)� nonterminal�� pgen_grammar�stackZ stack_noder r r �_flows_finished� s rT c C sB | j dkr| jd } | j dv r(| jd } | j dv o@| jd j dkS )N� decoratedr& �Z async_funcdef� async_stmt�ZclassdefZfuncdef�suite)r r r r r r �_func_or_class_has_suite� s rZ c C sL t | |�sdS t|�D ]0}|jdkr* dS |jdkrt|j�dk S qdS )NF� decoratorrY r T)rT �reversedrP r+ �nodesrQ r r r � _suite_or_file_input_is_valid� s r^ c C sB | j dkr| jd } z| jd j}W n ty8 Y dS 0 |dv S )NrW r r F)�if�for�while�try�with)r r r( r' )r r( r r r � _is_flow_node� s rd c @ s e Zd ZdS )�_PositionUpdatingFinishedN)�__name__� __module__� __qualname__r r r r re � s re c C sR | D ]H}z |j }W n, ty> | j|7 _||u r:t�Y q0 t|||� qd S r )r r' r1 re �_update_positions)r] � line_offsetrG r r r r r ri � s ri c @ sZ e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd d � Z dd� Z dd� Z ddd�Z dS )� DiffParserz� An advanced form of parsing a file faster. Unfortunately comes with huge side effects. It changes the given module. c C s || _ || _|| _d S r )� _pgen_grammar� _tokenizer�_module)�selfrR � tokenizerr@ r r r �__init__� s zDiffParser.__init__c C s d| _ d| _t| j�| _d S �Nr )� _copy_count� _parser_count� _NodesTreern � _nodes_tree�ro r r r �_reset s zDiffParser._resetc C s� t �d� d| j_|| _| �� t|�}t�d|| j�}|� � }t �dt|�|f � |D ]�\}}}} } t �d||d || d | � | |kr�|d dkr�| d8 } |dkr�| | } | � | |d || � qZ|d kr�| j | d � qZ|d kr�| j | d � qZ|d ksZJ �qZ| j � � t�r�zRd�|�} | j�� | k�s0J �t| j� t| jd d��| �|��} t| j| � W n( t�y� tt| j||�� � Y n0 | jjd }||k�r�td||f t| j||� ��t �d� | jS )a� The algorithm works as follows: Equal: - Assure that the start is a newline, otherwise parse until we get one. - Copy from parsed_until_line + 1 to max(i2 + 1) - Make sure that the indentation is correct (e.g. add DEDENT) - Add old and change positions Insert: - Parse from parsed_until_line + 1 to min(j2 + 1), hopefully not much more. Returns the new module node. zdiff parser startNz line_lengths old: %s; new: %sz!-> code[%s] old[%s:%s] new[%s:%s]r r& r: �equal�replace)� until_line�insert�deleteT�Zerror_recoveryr z (%s != %s) zdiff parser end) �LOG�debugrn Z _used_names�_parser_lines_newrx r+ r< ZSequenceMatcherZ get_opcodes�_copy_from_old_parser�_parserv �close�DEBUG_DIFF_PARSERr? r; r/ r rl �parserm r6 �AssertionError�printrC r* � Exception)ro rA rB � line_length�smZopcodesZ operation�i1�i2Zj1Zj2rj �codeZwithout_diff_parser_moduleZlast_posr r r �update s^ � �� �� zDiffParser.updatec C s2 | j �� d�|�kr.t�dd�|�d�|�� d S )Nr: zparser issue: %s %s)rn r; r? r �warning)ro rA Z lines_newr r r �_enabled_debuggingV s z DiffParser._enabled_debuggingc C s d}|| j jk�r| j j| }| �|d �}|d u rH| �| j jd � n�|jj}|�|�} |dkr||d �� j� t �r|g } n$| j jd } | j � || d � ||�} | r�| j d7 _ | j j} t �d| d jd | d jd d | | � n| �| j jd � || j jk�sJ |��| j j}qd S )Nr& r r zcopy old[%s:%s] new[%s:%s])rv �parsed_until_line�_get_old_line_stmtr� r. r �index�get_first_leafr) r, r � copy_nodesrs r r� r# r* ) ro rj Zstart_line_oldZuntil_line_oldZuntil_line_new�last_until_lineZparsed_until_line_oldZ line_stmtZ p_childrenr� Z copied_nodesZfrom_�tor r r r� Z s8 � � �z DiffParser._copy_from_old_parserc C sb | j j|dfdd�}t|�r$|�� }|�� d |kr^|}|jjdvrL|j}q8|jd |kr^|S d S )Nr T)Zinclude_prefixes)� file_inputrY )rn Zget_leaf_for_positionrF r r- r. r r# )ro Zold_liner r r r r r� � s z DiffParser._get_old_line_stmtc C s� d}|| j jkr�| �|�}|j}| j �|| j� | jdurH| j| j jd< t� d|d � � d | j j|j d d � || j jks�J |��| j j}qdS )zy Parses at least until the given line, but might just parse more until a valid state is reached. r Nr& z/parse_part from %s to %s (to %s in part parser)r ) rv r� �_try_parse_partr �add_parsed_nodes�_keyword_token_indents�_replace_tos_indent�indentsr r� r- r* )ro r{ r� r r] r r r r� � s � zDiffParser._parsec C sR | j d7 _ | jj}| j|d� }| j|||d�}t| jdd�| _| jj|d�S )z� Sets up a normal parser that uses a spezialized tokenizer to only parse until a certain position (or a bit longer if the statement hasn't ended. r N)rj Tr~ )�tokens) rt rv r� r� �_diff_tokenizer rl �_active_parserr� )ro r{ r� Z lines_afterr� r r r r� � s ��zDiffParser._try_parse_partr c c s� d}| j j}t|�}| j||d df||dkd�}| jj}d | _i | _|D �]T} | j} | t k�rt|�|k �rtt |� \} } } } } | t t fv r�| t kr�| d d | _qnq�qnd| v s�d| v r�t � dd| �} n4| d t| �ks�J t| ���| d t| � dkr�d} ttd| | �V �q�n^| tk�r6| jd |k�r6d }n>|�rtd}t|�|k�rtt| j|��rtttd| jd�V �q�| tk�r�| jd v �r�t|�| j| j< | V qJd S ) NFr r )r# r� Zis_first_tokenr$ r% z [^\n\r]+\Zr: T)�class�def)rv r� r+ rm r� rS r� r� r r �nextr �re�subr7 r � ENDMARKER�NEWLINEr# r^ rl �NAME�string�list)ro �linesr{ rj Z was_newliner� Zinitial_indentation_countr� rS �tokenrO r� r# r) r r r r� � sX � �zDiffParser._diff_tokenizeN)r ) rf rg rh �__doc__rq rx r� r� r� r� r� r� r� r r r r rk � s L,rk c @ sJ e Zd Zedd�Zddd�Zdd� Zd d � Zdd d �Zd d� Z dd� Z dS )�_NodesTreeNode�_ChildrenGroupz1prefix children line_offset last_line_offset_leafNr c C s" || _ g | _|| _g | _|| _d S r )r! �_children_groupsr. �_node_children� indentation)ro r! r. r� r r r rq s z_NodesTreeNode.__init__c C s� g }| j D ]\\}}}}t|d �� �}||j |_|dkr^zt|||� W n ty\ Y n0 ||7 }q || j_|D ] }| j|_qt| j D ] }|� � q�d S rr ) r� r r� r) ri re r! r r. r� �finish) ro r r) Z children_partrj �last_line_offset_leafZ first_leafr Z node_childr r r r� s& � � z_NodesTreeNode.finishc C s | j �|� d S r )r� �append)ro � child_noder r r �add_child_node( s z _NodesTreeNode.add_child_nodec C s4 |d u r|d � � }| �||||�}| j�|� d S �Nr& )rE r� r� r� )ro r) r rj r� �groupr r r �add_tree_nodes+ s �z _NodesTreeNode.add_tree_nodesc C s� d}| j r@| j d }t|j�}|jd |j }t||�r@|d8 }|tt|��d 7 }|rt|�d�st|�d�st|d7 }| j r�t || j d � |��S |S )Nr r& r r$ r% ) r� r r� r* rj rF r+ r rM r� �max� get_last_line)ro rN r1 Zchildren_grouprG r r r r� 4 s � z _NodesTreeNode.get_last_linec C s d| j j| jf S )Nz<%s: %s>)� __class__rf r! rw r r r �__repr__L s z_NodesTreeNode.__repr__)Nr )r N) rf rg rh r r� rq r� r� r� r� r� r r r r r� s � � r� c @ sb e Zd Zdd� Zedd� �Zdd� Zdd� Zd d � Zd d � Z d d� Z dd� Z ddd�Z dd� Z dS )ru c C s2 t |�| _| jg| _|| _d| _d| _dg| _d S )Nr: r )r� � _base_node�_working_stackrn �_prefix_remainderr) r� )ro r@ r r r rq Q s z_NodesTree.__init__c C s | j d �| j�S r� )r� r� r) rw r r r r� Y s z _NodesTree.parsed_until_linec C sB t t| j��D ].}|j|k s*|| jd u r2| S | j�� qd S rr )r\ r� r� r� �pop)ro r� r r r r �_update_insertion_node] s z!_NodesTree._update_insertion_nodec C sz | j }| �|�}|s$|| j | _ d S |d jdks6J �| �|d jd �}|jjdv sZJ �|�||� | �|d |� d S )Nr rK r )rY r� r& )r) �_remove_endmarkerr r� r# r! r� �_update_parsed_node_tos)ro � tree_nodes�keyword_token_indents� old_prefixr r r r r� c s z_NodesTree.add_parsed_nodesc C s� |j dkrn|jjd }t|||j d d�}|�dt|j�� | jd �|� | j� |� | � |jd |� nt |�r�| � |jd |� d S )NrY r r& �r� r: ) r r. r r� r# r� r� r� r� r� r� rZ )ro r! r� Zdef_leafZnew_tosr r r r� r s � z"_NodesTree._update_parsed_node_tosc C s� |d � � }|jdk}d| _|rn|j}t|�d�|�d��}|dkrn|jd|d � |j|d d� |_| _d| _|r�|j| _|dd� }|S )zE Helps cleaning up the tree nodes that get inserted. r& rD r: r$ r% Nr )rE r r� r) r� �rfind)ro r� rG Z is_endmarkerr) Z separationr r r r� � s "� z _NodesTree._remove_endmarkerc c sd t |�}|rt|�V t|�}t|�}|s8|| jvr8d S |V |D ] }t|�|krX d S |V qBd S r )�iterr� r r� )ro r� � is_new_suiteZ node_iteratorZ first_node�indentrH r r r �_get_matching_indent_nodes� s z%_NodesTree._get_matching_indent_nodesc s� |d j dv rg S t|d �� t| j�}| j}| j}� fdd�| jD �| _| �� � | �t| j�|||| j�\}| _| _}|r�| j|7 _n|| _|| _|| _|S )zy Copies tree nodes from the old parser tree. Returns the number of tree nodes that were copied. r �r � error_nodec s g | ]}|� kr|�qS r r )�.0�ir� r r � <listcomp>� � z)_NodesTree.copy_nodes.<locals>.<listcomp>)r r r� r� r) r� r� � _copy_nodes) ro r� r{ rj Zold_working_stackr� Z old_indents� new_nodes� added_indentsr r� r r� � s* �z_NodesTree.copy_nodesr: Fc C sF g }g }t | j||d��}d} |D ]�} | jd |kr: �q| jdkrJ �q| jdkrd| jdv rd �qt| �|kr�t| �r�|�| � �qz | j} W n t y� Y n^0 | } | jdkr�| jd } | jd v r�| jd } | jd v r�| jd } n| d } | jd v �r �q|�| � q"|�r�|�r�|d }|jd v �s<t |d ��rzd} |� � |�r|d }|� � jd k�rj�q|� � �qH�qt |�d k�r�|d jdk�r�|� � �q�q��q|�s�g |||fS |d }|d }d}t|��r�|}|jdk�r�|jd }�q�t|�}|�|� t|t|�d�}| j||g |j||dd�\}}} }||7 }t |�dk �rd|� � d} n |�snJ �|�|� |}d}|�r:t|d � � ��s�|�s�|d �� j}t|dd�d } |�r|d }|jdk�r�|jd }|jd v �r�|jd }|jd � � }|dk�s J �n |d � � }|�||||� | }d| _||||fS )N)r� r: r rD r )r r rU r& rV rX r� rK r ���r� FrY r� T)� is_nested� r8 �:)r� r� r# r r rI rZ r� r r' rd r� rE r+ r" r� r r� r� rF r r) r r� r� )ro Z working_stackr] r{ rj r) r� r� r� � new_prefixr �crH Z suite_nodeZ last_nodeZtosZhad_valid_suite_lastrY r� Z suite_tosZ suite_nodesZnew_working_stackZai�p�lastr� r r r r� � s� � � � �z_NodesTree._copy_nodesc C s | j �� z| j�� }W n ty2 ddg}Y n0 t|�}t|j�}t| j �}t |�dks`J �t |�dkr�|d � t �r�|ddgkr�|d d8 < |d t |d �7 < n(|d t |�d 7 < t |d �|d< t dt|�| j | j �}| j|_| jj�|� d S )Nr r r& r: )r� r� rn rE � IndexErrorrJ r� r* r r) r+ r, r r �tupler� r. r r� )ro rG r* r� rD r r r r� _ s$ z_NodesTree.closeN)r: F)rf rg rh rq �propertyr� r� r� r� r� r� r� r� r� r r r r ru P s # � ru )r: )0r� r� r< � collectionsr �loggingZ parso.utilsr Zparso.python.parserr Zparso.python.treer Zparso.python.tokenizer r Zparso.python.tokenr � getLoggerrf r r� r r� r r� r r� r r r r" r r/ r6 rC rI rJ rF rT rZ r^ rd r� re ri rk r� ru r r r r �<module> sN . K
| ver. 1.1 | |
.
| PHP 8.3.30 | Ð“ÐµÐ½ÐµÑ€Ð°Ñ†Ð¸Ñ Ñтраницы: 0 |
proxy
|
phpinfo
|
ÐаÑтройка