o g;4@sdZddlZddlZddlZddlZddlmZddlmZddlm Z m Z ddl m Z ddl Z ddlZddlmZddlmZmZmZGd d d Zd d Zed krfdZdZdZeedZeeeZdSdS)z+Module for interacting with S3 bucket filesN)datetime)BytesIO)IOList)urlparse) ClientError) backoff_hdlr failure_hdlr success_hdlrc @sJeZdZdZd-deddfddZejeje de e e d  d.d ed ed efd dZ dedefddZejeje de e e d  d/d ededefddZejeje de e e d d0ddZejeje de e e d d.ded efddZ d1dededeeeffdd Zded!edeefd"d#Zd2d%ed&ed'efd(d)Zd*efd+d,ZdS)3S3BucketStoragezb Class to handle operations related to an S3 bucket, such as uploading and getting files. xcap-bucket bucket_namereturnNcCstd|_||_dS)z Initializes the S3BucketStorage class with the specified bucket name. Args: bucket_name (str): The name of the S3 bucket. s3N)boto3client s3_clientr )selfr r,/var/www/html/XCapMarket/utils/s3_storage.py__init__s  zS3BucketStorage.__init__) max_tries on_backoff on_success on_giveup s3_file_pathsource_file_paths3_filec Cs<d|vr tj|}n|}zf|s;|r;|r;t|d}|j||j|||j||_Wdn1s5wYn+|rR|rR|jj ||j|d}||j||_n|rbtj |sbt d|dt dd|d|d |jd WSt y}zt d td ||j|t d |d|jd||d}~ww)a( Uploads a file to the specified location in the S3 bucket with retry logic. Args: s3_file_path (str): The path to upload the file to in the S3 bucket. source_file_path (str): The path of the file to upload from the local system. max_retries (int): Maximum number of retries if upload fails (default is 5). retry_delay (int): Delay between retries in seconds (default is 2). Returns: str: A message indicating the success or failure of the upload operation. /rbN)BodyBucketKeyzFile z( does not exist or file is not provided.z4Enter source file path or base64 encode binary file.z uploaded to  in z bucket.z>Error %s: Failed uploading file %s to S3 bucket %s. Error %s%Y/%m/%d %H:%M:%SzFailed uploading z to z bucket: )ospathbasenameopenrupload_fileobjr create_object_bucket_urls3_file_location put_objectexists ValueErrorloggingerrorrnowstrftime)rrrr file_name source_fileresponseverrrupload_file_to_s3_bucket"sT    z(S3BucketStorage.upload_file_to_s3_bucket object_keycCsd|d|S)zCreate the object URL for S3zhttps://z.s3.amazonaws.com/r)rr r9rrrr+isz(S3BucketStorage.create_object_bucket_url max_retries retry_delayc CszE|drt|}|j|_|jd}|jj|j|d}|dddkr;|d}|s8t d|d |j|WSt d |jd |t yk}zt d t d ||j|t d|d||d}~ww)a Retrieves a file from the specified location in the S3 bucket with retry logic. Args: s3_file_path (str): The path of the file to retrieve from the S3 bucket. max_retries (int): Maximum number of retries if retrieval fails (default is 5). retry_delay (int): Delay between retries in seconds (default is 2). Returns: bytes: The content of the retrieved file as binary data. s3://rr"r#ResponseMetadataHTTPStatusCoder!zFile content is empty at r$zError getting object from z. File not found at zCError %s Error: Empty file content %s from S3 bucket %s. Error: %sr%zFailed getting z: N) startswithrnetlocr r'lstripr get_objectreadr/r0r1rr2r3)rrr;r< parsed_urir6 file_contentr7rrrget_file_from_s3_bucketms:    z'S3BucketStorage.get_file_from_s3_bucketc CszDg}|jj|jd}|dgD]}|d}d|vr0|dddd}||vr0||qt|dkrBtd|jd ||WStyn}zt d t d |j||td |jd |d||d}~ww)a Search for files containing the specified keyword in the S3 bucket with retry logic. Args: keyword (str): The keyword to search for in file names. max_retries (int): Maximum number of retries if retrieval fails (default is 5). retry_delay (int): Delay between retries in seconds (default is 2). Returns: list: A list of file paths in the bucket that contain the keyword. )r"Contentsr#.rrzNo files in S3 bucket z has keyword zLError %s: Error searching files in S3 bucket %s with keyword '%s'. Error:%sr%z#Error searching files in S3 bucket z with keyword 'z': N) rlist_objects_v2r getsplitappendlenr/r0r1rr2r3) rkeywordr;matching_filesr6objfile_keyr4r7rrrsearch_files_in_buckets<   z&S3BucketStorage.search_files_in_bucket source_folderc Csz0|s|}t|D]}tjtj||r)tj||}||}|j||d}q d|dWStyM}ztd|tdd|i|d}~ww)a Bulk File Upload method to perform upload and download operations with the S3 bucket. Args: source_folder (str): The path of the local folder containing files to upload. s3_file_path (str): The path of folder to upload files to in the S3 bucket. Returns: bytes: The content of the retrieved file from the S3 bucket. )rrzFiles uploaded to z successfully.zError: messageN)r&listdirr'isfilejoinr8r/print)rrWrfilerr6r7rrrbulk_file_uploads"z S3BucketStorage.bulk_file_uploads3_key expires_inc Cslddl}z||\}}|jjd|j|d|pdd|d}||fWSty5}ztd||d}~ww) a Generates a signed URL for an S3 object. Args: s3_key (str): The S3 object key expires_in (int): URL expiration time in seconds (default: 1 hour) Returns: tuple[str, str]: A tuple containing (signed_url, mime_type) rNrEzinline; filename=document.pdfzapplication/pdf)r"r#ResponseContentDispositionResponseContentType) ClientMethodParams ExpiresInzError generating signed URL: ) mimetypes guess_typergenerate_presigned_urlr rr/)rr`rarg mime_type_ signed_urlerrrgenerate_signed_urls"  z#S3BucketStorage.generate_signed_url folder_namecCsF|jj||d}g}|dgD]}|d|krq||dq|S)zList all the files in an S3 folder. Args: bucket_name: Name of the S3 bucket folder_name: Path to the folder in the bucket Returns: List of file paths )r"PrefixrJr#)rrMrNrP)rr ror6filesrTrrrlist_files_in_folder*s  z$S3BucketStorage.list_files_in_folder project_idteam_id nested_pathcCs|d|d}|r||d}tjd}|||}g}|D].}d|d|} tj|} tj| d} |jj||d} | d} || | | | dq |S)zGet the files in a folderrxcap_s3_storager=r> ContentLength)r4file_url file_type file_size) r&environrrr'r(splitextr head_objectrP)rrtrurvror rq file_detailsrUrzr4r{r6r|rrrget_files_in_folderAs(    z#S3BucketStorage.get_files_in_folder file_pathcCs|jj|j|ddS)zDelete the file from S3r>T)r delete_objectr )rrrrrdelete_file_from_s3^sz#S3BucketStorage.delete_file_from_s3)r )NN)Nrr:)r)r_)rs)__name__ __module__ __qualname____doc__strrbackoff on_exceptionexporrr r r8r+intrIrVr^tuplernrrrrrrrrrr s   ?  7 )!  %r cCs tjdS)zGet the default bucket namerw)r&r}rrrrget_default_bucket_nameds r__main__zxcap-storage-devzimages/zgeneric/images/unity/)r )rbase64r0r&timeriortypingrr urllib.parserrrbotocore.exceptionsrutils.backoff_handlersrr r r rr BUCKET_NAME SOURCE_FOLDER S3_FILE_PATHs3_storage_objr^r6rrrrs0    S