Файловый менеджер - Редактировать - /opt/imh-python/lib/python3.9/site-packages/s3transfer/__pycache__/upload.cpython-39.pyc
Ðазад
a s�h�s � @ s d dl Z d dlmZ d dlmZmZ d dlmZ d dlm Z d dlm Z d dlm Z d dlm Z d d l mZ d d l mZ d d l mZmZ G d d � d e�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de �ZG dd� de �ZG d d � d e �ZdS ) � N)�six)�seekable�readable)�IN_MEMORY_UPLOAD_TAG)�Task)�SubmissionTask)�CreateMultipartUploadTask)�CompleteMultipartUploadTask)� get_callbacks)�get_filtered_dict)�DeferredOpenFile�ChunksizeAdjusterc @ s. e Zd Zd dd�Zdd� Zdd� Zdd � Zd S ) �AggregatedProgressCallback� c C s || _ || _d| _dS )a� Aggregates progress updates for every provided progress callback :type callbacks: A list of functions that accepts bytes_transferred as a single argument :param callbacks: The callbacks to invoke when threshold is reached :type threshold: int :param threshold: The progress threshold in which to take the aggregated progress and invoke the progress callback with that aggregated progress total r N)� _callbacks� _threshold� _bytes_seen)�self� callbacks� threshold� r ��/root/rpmbuild/BUILDROOT/imh-python39-modules-3.9.7-92.el8.x86_64/opt/imh-python/lib/python3.9/site-packages/s3transfer/upload.py�__init__ s z#AggregatedProgressCallback.__init__c C s& | j |7 _ | j | jkr"| �� d S �N)r r �_trigger_callbacks)r �bytes_transferredr r r �__call__- s z#AggregatedProgressCallback.__call__c C s | j dkr| �� dS )z@Flushes out any progress that has not been sent to its callbacksr N)r r �r r r r �flush2 s z AggregatedProgressCallback.flushc C s" | j D ]}|| jd� qd| _d S )N)r r )r r )r �callbackr r r r 7 s z-AggregatedProgressCallback._trigger_callbacksN)r )�__name__� __module__� __qualname__r r r r r r r r r s r c @ sL e Zd ZdZdd� Zddd�Zddd �Zd d � Zd d � Zdd� Z dd� Z dS )�InterruptReadera� Wrapper that can interrupt reading using an error It uses a transfer coordinator to propagate an error if it notices that a read is being made while the file is being read from. :type fileobj: file-like obj :param fileobj: The file-like object to read from :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator to use if the reader needs to be interrupted. c C s || _ || _d S r )�_fileobj�_transfer_coordinator)r �fileobj�transfer_coordinatorr r r r J s zInterruptReader.__init__Nc C s | j jr| j j�| j�|�S r )r% � exceptionr$ �read)r �amountr r r r) N s zInterruptReader.readr c C s | j �||� d S r )r$ �seek)r �where�whencer r r r+ X s zInterruptReader.seekc C s | j �� S r )r$ �tellr r r r r. [ s zInterruptReader.tellc C s | j �� d S r )r$ �closer r r r r/ ^ s zInterruptReader.closec C s | S r r r r r r � __enter__a s zInterruptReader.__enter__c O s | � � d S r )r/ )r �args�kwargsr r r �__exit__d s zInterruptReader.__exit__)N)r ) r r! r" �__doc__r r) r+ r. r/ r0 r3 r r r r r# = s r# c @ sf e Zd ZdZddd�Zedd� �Zdd� Zd d � Zd d � Z d d� Z dd� Z dd� Z dd� Z dd� ZdS )�UploadInputManageraJ Base manager class for handling various types of files for uploads This class is typically used for the UploadSubmissionTask class to help determine the following: * How to determine the size of the file * How to determine if a multipart upload is required * How to retrieve the body for a PutObject * How to retrieve the bodies for a set of UploadParts The answers/implementations differ for the various types of file inputs that may be accepted. All implementations must subclass and override public methods from this class. Nc C s || _ || _|| _d S r )�_osutilr% �_bandwidth_limiter�r �osutilr' �bandwidth_limiterr r r r w s zUploadInputManager.__init__c C s t d��dS )a Determines if the source for the upload is compatible with manager :param upload_source: The source for which the upload will pull data from. :returns: True if the manager can handle the type of source specified otherwise returns False. z must implement _is_compatible()N��NotImplementedError��clsZ upload_sourcer r r � is_compatible| s z UploadInputManager.is_compatiblec C s t d��dS )a� Whether the body it provides are stored in-memory :type operation_name: str :param operation_name: The name of the client operation that the body is being used for. Valid operation_names are ``put_object`` and ``upload_part``. :rtype: boolean :returns: True if the body returned by the manager will be stored in memory. False if the manager will not directly store the body in memory. z%must implement store_body_in_memory()N)�NotImplemented�r �operation_namer r r �stores_body_in_memory� s z(UploadInputManager.stores_body_in_memoryc C s t d��dS )z�Provides the transfer size of an upload :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request z&must implement provide_transfer_size()Nr; �r �transfer_futurer r r �provide_transfer_size� s z(UploadInputManager.provide_transfer_sizec C s t d��dS )a� Determines where a multipart upload is required :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: boolean :returns: True, if the upload should be multipart based on configuartion and size. False, otherwise. z*must implement requires_multipart_upload()Nr; �r rE �configr r r �requires_multipart_upload� s z,UploadInputManager.requires_multipart_uploadc C s t d��dS )a� Returns the body to use for PutObject :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: s3transfer.utils.ReadFileChunk :returns: A ReadFileChunk including all progress callbacks associated with the transfer future. z$must implement get_put_object_body()Nr; rD r r r �get_put_object_body� s z&UploadInputManager.get_put_object_bodyc C s t d��dS )a Yields the part number and body to use for each UploadPart :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type chunksize: int :param chunksize: The chunksize to use for this upload. :rtype: int, s3transfer.utils.ReadFileChunk :returns: Yields the part number and the ReadFileChunk including all progress callbacks associated with the transfer future for that specific yielded part. z)must implement yield_upload_part_bodies()Nr; )r rE � chunksizer r r �yield_upload_part_bodies� s z+UploadInputManager.yield_upload_part_bodiesc C s* t || j�}| jr&| jj|| jdd�}|S )NF)�enabled)r# r% r7 Zget_bandwith_limited_stream)r r& r r r � _wrap_fileobj� s �z UploadInputManager._wrap_fileobjc C s t |d�}|rt|�gS g S )N�progress)r r )r rE r r r r �_get_progress_callbacks� s z*UploadInputManager._get_progress_callbacksc C s dd� |D �S )Nc S s g | ] }|j �qS r )r )�.0r r r r � <listcomp>� � z;UploadInputManager._get_close_callbacks.<locals>.<listcomp>r )r Z aggregated_progress_callbacksr r r �_get_close_callbacks� s z'UploadInputManager._get_close_callbacks)N)r r! r" r4 r � classmethodr? rC rF rI rJ rL rN rP rT r r r r r5 h s r5 c @ sd e Zd ZdZedd� �Zdd� Zdd� Zdd � Zd d � Z d d � Z dd� Z dd� Z dd� Z dd� ZdS )�UploadFilenameInputManagerz Upload utility for filenamesc C s t |tj�S r )� isinstancer � string_typesr= r r r r? � s z(UploadFilenameInputManager.is_compatiblec C s dS )NFr rA r r r rC � s z0UploadFilenameInputManager.stores_body_in_memoryc C s |j �| j�|j jj�� d S r )�metarF r6 Z get_file_size� call_argsr& rD r r r rF � s ��z0UploadFilenameInputManager.provide_transfer_sizec C s |j j|jkS r )rY �size�multipart_thresholdrG r r r rI � s z4UploadFilenameInputManager.requires_multipart_uploadc C sJ | � |�\}}| �|�}| �|�}| �|�}|jj}| jj|||||d�S )N�r& � chunk_size�full_file_sizer �close_callbacks)�&_get_put_object_fileobj_with_full_sizerN rP rT rY r[ r6 �#open_file_chunk_reader_from_fileobj)r rE r& � full_sizer r` r[ r r r rJ � s � �z.UploadFilenameInputManager.get_put_object_bodyc c s� |j j}| �||�}td|d �D ]j}| �|�}| �|�}||d }| j|j jj|||d�\} } | � | �} | j j | || ||d�} || fV q"d S )N� )� start_byte� part_sizer_ r] ) rY r[ �_get_num_parts�rangerP rT �'_get_upload_part_fileobj_with_full_sizerZ r& rN r6 rb ) r rE rK r_ Z num_parts� part_numberr r` re r& rc Zread_file_chunkr r r rL s" � �z3UploadFilenameInputManager.yield_upload_part_bodiesc C s t ||| jjd�}|S )N)Z open_function)r r6 �open)r r& re r r r �_get_deferred_open_file s �z2UploadFilenameInputManager._get_deferred_open_filec C s" |j jj}|j j}| �|d�|fS )Nr )rY rZ r& r[ rl �r rE r& r[ r r r ra # s zAUploadFilenameInputManager._get_put_object_fileobj_with_full_sizec K s |d }|d }| � ||�|fS )Nre r_ )rl )r r& r2 re rc r r r ri ( s zBUploadFilenameInputManager._get_upload_part_fileobj_with_full_sizec C s t t�|jjt|� ��S r )�int�math�ceilrY r[ �float)r rE rf r r r rg - s �z)UploadFilenameInputManager._get_num_partsN)r r! r" r4 rU r? rC rF rI rJ rL rl ra ri rg r r r r rV � s rV c @ s< e Zd ZdZedd� �Zdd� Zdd� Zdd � Zd d � Z d S ) �UploadSeekableInputManagerz&Upload utility for an open file objectc C s t |�ot|�S r )r r r= r r r r? 4 s z(UploadSeekableInputManager.is_compatiblec C s |dkr dS dS d S )N� put_objectFTr rA r r r rC 8 s z0UploadSeekableInputManager.stores_body_in_memoryc C sD |j jj}|�� }|�dd� |�� }|�|� |j �|| � d S )Nr � )rY rZ r& r. r+ rF )r rE r& Zstart_positionZ end_positionr r r rF > s �z0UploadSeekableInputManager.provide_transfer_sizec K s |� |d �}t�|�t|�fS )Nrf )r) r �BytesIO�len)r r& r2 �datar r r ri J s zBUploadSeekableInputManager._get_upload_part_fileobj_with_full_sizec C s" |j jj}|�� |j j }||fS r )rY rZ r& r. r[ rm r r r ra Y s zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizeN) r r! r" r4 rU r? rC rF ri ra r r r r rr 2 s rr c sh e Zd ZdZd� fdd� Zedd� �Zdd� Zd d � Zd d � Z d d� Z dd� Z ddd�Z dd� Z � ZS )� UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.Nc s t t| ��|||� d| _d S )NrS )�superrx r � _initial_datar8 �� __class__r r r c s �z&UploadNonSeekableInputManager.__init__c C s t |�S r )r r= r r r r? h s z+UploadNonSeekableInputManager.is_compatiblec C s dS )NTr rA r r r rC l s z3UploadNonSeekableInputManager.stores_body_in_memoryc C s d S r r rD r r r rF o s z3UploadNonSeekableInputManager.provide_transfer_sizec C sT |j jd ur|j j|jkS |j jj}|j}| �||d�| _t| j�|k rLdS dS d S )NFT)rY r[ r\ rZ r& �_readrz rv )r rE rH r& r r r r rI t s z7UploadNonSeekableInputManager.requires_multipart_uploadc C s@ | � |�}| �|�}|jjj}| �| j|�� ||�}d | _|S r )rP rT rY rZ r& � _wrap_datarz r) )r rE r r` r& �bodyr r r rJ � s �z1UploadNonSeekableInputManager.get_put_object_bodyc c s^ |j jj}d}| �|�}| �|�}|d7 }| �||�}|s<qZ| �|||�}d }||fV qd S )Nr rd )rY rZ r& rP rT r} r~ ) r rE rK Z file_objectrj r r` Z part_contentZ part_objectr r r rL � s �z6UploadNonSeekableInputManager.yield_upload_part_bodiesTc C sx t | j�dkr|�|�S |t | j�krL| jd|� }|rH| j|d� | _|S |t | j� }| j|�|� }|rtd| _|S )a= Reads a specific amount of data from a stream and returns it. If there is any data in initial_data, that will be popped out first. :type fileobj: A file-like object that implements read :param fileobj: The stream to read from. :type amount: int :param amount: The number of bytes to read from the stream. :type truncate: bool :param truncate: Whether or not to truncate initial_data after reading from it. :return: Generator which generates part bodies from the initial data. r NrS )rv rz r) )r r& r* �truncaterw Zamount_to_readr r r r} � s z#UploadNonSeekableInputManager._readc C s. | � t�|��}| jj|t|�t|�||d�S )a� Wraps data with the interrupt reader and the file chunk reader. :type data: bytes :param data: The data to wrap. :type callbacks: list :param callbacks: The callbacks associated with the transfer future. :type close_callbacks: list :param close_callbacks: The callbacks to be called when closing the wrapper for the data. :return: Fully wrapped data. r] )rN r ru r6 rb rv )r rw r r` r& r r r r~ � s �z(UploadNonSeekableInputManager._wrap_data)N)T)r r! r" r4 r rU r? rC rF rI rJ rL r} r~ � __classcell__r r r{ r rx a s *rx c @ sX e Zd ZdZg d�ZdgZdd� Zddd�Zd d � Zd d � Z d d� Z dd� Z dd� Z dS )�UploadSubmissionTaskz.Task for submitting tasks to execute an upload)ZSSECustomerKeyZSSECustomerAlgorithmZSSECustomerKeyMD5� RequestPayerr� c C sH t ttg}|jjj}|D ]}|�|�r| S qtd|t|�f ��dS )ao Retrieves a class for managing input for an upload based on file type :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future for the request :rtype: class of UploadInputManager :returns: The appropriate class to use for managing a specific type of input for uploads. z&Input %s of type: %s is not supported.N) rV rr rx rY rZ r& r? � RuntimeError�type)r rE Z upload_manager_resolver_chainr& Zupload_manager_clsr r r � _get_upload_input_manager_cls� s � ��z2UploadSubmissionTask._get_upload_input_manager_clsNc C sd | � |�|| j|�}|jjdu r*|�|� |�||�sL| �||||||� n| �||||||� dS )a� :param client: The client associated with the transfer manager :type config: s3transfer.manager.TransferConfig :param config: The transfer config associated with the transfer manager :type osutil: s3transfer.utils.OSUtil :param osutil: The os utility associated to the transfer manager :type request_executor: s3transfer.futures.BoundedExecutor :param request_executor: The request executor associated with the transfer manager :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for N)r� r% rY r[ rF rI �_submit_upload_request�_submit_multipart_request)r �clientrH r9 �request_executorrE r: �upload_input_managerr r r �_submit s$ �� � � �z UploadSubmissionTask._submitc C sN |j j}| �|d�}| jj|t| j||�|�|j|j|j d�dd�|d� d S )Nrs )r� r& �bucket�key� extra_argsT)r' � main_kwargs�is_final��tag) rY rZ �_get_upload_task_tagr% �submit� PutObjectTaskrJ r� r� r� ) r r� rH r9 r� rE r� rZ Zput_object_tagr r r r� 2 s( ���� �z+UploadSubmissionTask._submit_upload_requestc C s� |j j}| j�|t| j||j|j|jd�d��}g } | �|j�} | � |d�} |j j } t � } | � |j | �}|�||�}|D ]B\}}| �| jj|t| j|||j|j|| d�d|id�| d�� qx| �|j�}| j�|t| j||j|j|d�|| d�d d �� d S ) N)r� r� r� r� )r' r� � upload_part)r� r& r� r� rj r� � upload_id)r' r� �pending_main_kwargsr� )r� �partsT)r' r� r� r� )rY rZ r% r� r r� r� r� �_extra_upload_part_argsr� r[ r Zadjust_chunksizeZmultipart_chunksizerL �append�UploadPartTask� _extra_complete_multipart_argsr )r r� rH r9 r� rE r� rZ Zcreate_multipart_futureZ part_futuresZextra_part_argsZupload_part_tagr[ ZadjusterrK Z part_iteratorrj r& Z complete_multipart_extra_argsr r r r� L sx ��� �� � ���������z.UploadSubmissionTask._submit_multipart_requestc C s t || j�S r )r �UPLOAD_PART_ARGS�r r� r r r r� � s z,UploadSubmissionTask._extra_upload_part_argsc C s t || j�S r )r �COMPLETE_MULTIPART_ARGSr� r r r r� � s z3UploadSubmissionTask._extra_complete_multipart_argsc C s d }|� |�rt}|S r )rC r )r r� rB r� r r r r� � s z)UploadSubmissionTask._get_upload_task_tag)N) r r! r" r4 r� r� r� r� r� r� r� r� r� r r r r r� � s � � 'Mr� c @ s e Zd ZdZdd� ZdS )r� z Task to do a nonmultipart uploadc C s@ |�(}|j f |||d�|�� W d � n1 s20 Y dS )aP :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param extra_args: A dictionary of any extra arguments that may be used in the upload. )�Bucket�Key�BodyN)rs )r r� r& r� r� r� r r r r �_main� s zPutObjectTask._mainN�r r! r" r4 r� r r r r r� � s r� c @ s e Zd ZdZdd� ZdS )r� z+Task to upload a part in a multipart uploadc C sR |�,}|j f |||||d�|��} W d � n1 s60 Y | d } | |d�S )a� :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param upload_id: The id of the upload :param part_number: The number representing the part of the multipart upload :param extra_args: A dictionary of any extra arguments that may be used in the upload. :rtype: dict :returns: A dictionary representing a part:: {'Etag': etag_value, 'PartNumber': part_number} This value can be appended to a list to be used to complete the multipart upload. )r� r� ZUploadId� PartNumberr� N�ETag)r� r� )r� ) r r� r& r� r� r� rj r� r �response�etagr r r r� � s ��$zUploadPartTask._mainNr� r r r r r� � s r� ) ro Zbotocore.compatr Zs3transfer.compatr r Zs3transfer.futuresr Zs3transfer.tasksr r r r Zs3transfer.utilsr r r r �objectr r# r5 rV rr rx r� r� r� r r r r �<module> s* !+yQ/ D
| ver. 1.1 | |
.
| PHP 8.3.30 | Ð“ÐµÐ½ÐµÑ€Ð°Ñ†Ð¸Ñ Ñтраницы: 0 |
proxy
|
phpinfo
|
ÐаÑтройка