o tBh@s dZddlZddlZddlZddlZddlZddlmZddlm Z ddlm Z ddl m Z ddl mZddl Z ddlZddlmZddlZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZddlmZmZmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&zddl'Z'Wn e(ydZ'Ynwzddl)m*Z*Wn e(ydZ*Ynwzddl+Z+Wn e(ydZ+Ynwddl,m-Z-ddl,m.Z.ddl,m/Z/ddl,m0Z0ddl,m1Z1ddl,m2Z2ddl3m4Z4m5Z5ddl3m6Z6m7Z7ddl3m8Z8m9Z9ddl3m:Z:m;Z;mm?Z?ddl>m@Z@dgeAe;BZCeCdd e;D7ZCd!d"gZDeDd#gZEeFe:d$r9eCGe:Hd%e;e<ZId&d'ZJd(d)ZKd*d+ZLGd,d-d-eMZNd%d/d0ZOd1d2ZPd&d3d4ZQd5d6ZRd7d8ZSd9d:ZTd;d<ZUejVjd=d>d?e8d?d@fdAggdBdCedDdEZWedFeCedGgdHedIgdJdKdLZXedFeCdMdNZYdOdPZZeedQgdRdSdTZ[eedFgdUdVdWZ\dXdYZ]edZe;ed[e;d\d]Z^d^d_Z_d`daZ`dbdcZaedGgdddedfZbedFeEdgdhZcedidjZdedFeEdkdlZeeedFeEdmdnZfeedFeDdodpZgdqdrZhedFe;edsdgdtfdugdvfgdwdxZieedFeEdydzZjd{d|ZkeedFeDd}d~ZlddZmddZnddZoeedFeEejVjpddddZqGddde-ZrddZsedFeCddZtddZuddZvddZweEddZxexydd ezdDeedFexddZ{Gddde-Z|ddZ}ddZ~ddZddZGdddejj1Ze6deeedFgdddZeedGgdedFeEddZddZeedeEedeEddZeddZddZddZedgdddZedgdddZeddZeeedFeDddÄZdZeedFeDddƄZdZdZdZdZeedFeDe:durbgnd%gedeeegedgd͢ddτZdjejejejdэZeddӄZdjejejejdэZeddքZdd؄ZddڄZeeedFeDdd܄ZddބZeeddZddZedFeCedGgdddZeeedFeDddZedFgdddZeedFeDddZddZddZddZddZddZd'ddZeedFd"d#gddZee e+duddddZd(ddZedFd"d#gddZddZedFdd"ge e*duddddZe e+dudddd Zd d Zd d ZddZddZddZeeedGgdddZeeedgdedGdd?gddZeedGdd?gedgdddZeeedFd!d#e/e.gd d!ZeedGgd"d#d$ZdS()z Test the parallel module. N)format_exceptionsqrt)sleep) PicklingError TimeoutError)reloadparallel)dumpload)get_reusable_executor)np with_numpy)with_multiprocessing) parametrizeraisescheck_subprocess_callskipifSkipTestwarns)TerminatedWorkerError)Queue) parallel_sum)SequentialBackend)ThreadingBackend)MultiprocessingBackend)ParallelBackendBase) LokyBackend) SafeFunction)Paralleldelayed)register_parallel_backendparallel_backend)effective_n_jobs cpu_count)mpBACKENDSDEFAULT_BACKENDEXTERNAL_BACKENDS)JoblibException)WorkerInterruptcCsg|]}t|qS)r)).0 backend_strr.r.p/var/www/html/riverr-enterprise-integrations-main/venv/lib/python3.10/site-packages/joblib/test/test_parallel.py Gsr2multiprocessingloky threading get_contextspawncCst|dt|ddS)N_pool_workers)getattrbackendr.r.r1 get_workersRsr=cCs||SNr.)xyr.r.r1divisionVrAcC|dSNr.r?r.r.r1squareZrBrGc@seZdZdZddZdS)MyExceptionWithFinickyInitz1An exception class with non trivial __init__ cCdSr>r.)selfabcdr.r.r1__init__az#MyExceptionWithFinickyInit.__init__N__name__ __module__ __qualname____doc__rOr.r.r.r1rH^s rHFcCs"|dkr|r tddddt|S)NrKrLrMrN)rH ValueError)r?custom_exceptionr.r.r1exception_raiseres rYcCstdt)Ng?)timerKeyboardInterruptrFr.r.r1interrupt_raiserls r\cCs|d||S)zO A module-level function so that it can be spawn with multiprocessing. rEr.)r?r@zr.r.r1fqsr^cCsttdSNr)typer get_active_backendr.r.r.r1_active_backend_typexsrbcCst||dddtdDS)Nn_jobsr<cs|] }tt|VqdSr>r#rGr/ir.r.r1 }  z parallel_func..r"range) inner_n_jobsr<r.r.r1 parallel_func|srocCtdksJdSr_)r'r.r.r.r1test_cpu_countrqcCrpr_)r&r.r.r.r1test_effective_n_jobsrrrszbackend_n_jobs, expected_n_jobs)rkrkrdNr)z positive-intz negative-intNone)idscCsTtd|dtdd|ksJWdn1swYtdddks(JdS)Nr5rur)r%r&)backend_n_jobsexpected_n_jobsr.r.r1test_effective_n_jobs_Nones r{r<rd)rrErtverbose)rE dcCs:ddtdDt|||dddtdDksJdS)NcSg|]}t|qSr.rGr/r?r.r.r1r2z(test_simple_parallel..)rdr<r}csrer>rfrr.r.r1rirjz'test_simple_parallel..rmr")r<rdr}r.r.r1test_simple_parallels rcCs|jtdddtd}td|dddtdD}|gd ks&JWdn1s0wYd d |D}t|d ksDJdS) Nname!some_new_name_for_the_main_thread)targetrvaluerErccsrer>rfrr.r.r1rirjz6test_main_thread_renamed_no_warning..rk)rrcSsg|] }dt|jvr|qS)zworker timeout)strmessage)r/wr.r.r1r2z7test_main_thread_renamed_no_warning..r)setattrr5current_threadrr"rmlen)r< monkeypatchwarninforesultsr.r.r1#test_main_thread_renamed_no_warnings  rcCsptd}t||dWdn1swY|r.t|dkr,d|djjdvSdSt|dks6JdS)N)r<rnrzbacked parallel loops cannotFT)rrorrargs)r<rnexpectedrecordsr.r.r1_assert_warning_nesteds  rz%parent_backend,child_backend,expected))r4r3T)r4r4F)r3r3T)r3r4T)r5r3T)r5r4Tcsntd|dfddtdDtd|dfddtdD}|dkr/t|s-JdSt|s5JdS)NrErcc3s"|] }ttdddVqdS)rFr<rnrNr#rr/_ child_backendr.r1ri z0test_nested_parallel_warnings..rc3s"|] }ttddVqdS)rErNrrrrr.r1rirr5)r"rmanyall)parent_backendrrresr.rr1test_nested_parallel_warningss r)r4r3r5cCs>dg}dd}tj||fd}|||dsJdS)NFcSsdtd}tddddtdDWdn1swYtt|t|dk|d<dS)NrErucs|] }ttdVqdS)皙?Nr#rrr.r.r1rirjzPtest_background_thread_parallelism..background_thread..rr)rr"rmprintr)is_run_parallelrr.r.r1background_threads   z=test_background_thread_parallelism..background_thread)rrr)r5Threadstartjoin)r<rrtr.r.r1"test_background_thread_parallelisms rcCs"td|dddtdDdS)NrErccsr){Gz?Nrfrr.r.r1rirjznested_loop..rlr;r.r.r1 nested_loops rrrcs&td|dfddtdDdS)NrErcc3|] }ttVqdSr>)r#rrrr.r1rirjz#test_nested_loop..rl)rrr.rr1test_nested_loops rcCstr>)rWr;r.r.r1raise_exception rPrc Cstt5tddd}|ttdttdgWdn1s$wYWddSWddS1s.N)rr"rmfullr.r.rr1test_mutate_input_with_threadss r)rrErkcCs6td}dd|Dt|ddd|DksJdS)z.Check the keyword argument processing of pmap. cSg|]}t|ddqSrr@r^rr.r.r1r2#z(test_parallel_kwargs..rucs |] }tt|ddVqdSrrNr#r^rr.r.r1ri$z'test_parallel_kwargs..Nr)rdlstr.r.r1test_parallel_kwargss  rcCstd}dd|D}td|d4}|j}||dd|Dks"J||dd|Dks/Jtdur>t|t|jus>JWdn1sHwYtdurZt|jdusZJ||d d|DksgJtdurtt|jdusvJdSdS) NrcSrrrrr.r.r1r2*rz4test_parallel_as_context_manager..rrccsrrrrr.r.r1ri3rz3test_parallel_as_context_manager..csrrrrr.r.r1ri4rcsrrrrr.r.r1ri@r)rmr"_backendr(r=)r<rrpmanaged_backendr.r.r1 test_parallel_as_context_manager's"rcsbGdddtttddtddfddtd DWd d S1s*wYd S) ze Check that pmap captures the errors when it is passed an object that cannot be pickled. c@seZdZddZdS)z1test_parallel_pickling..UnpicklableObjectcStd)N123) RuntimeErrorrJr.r.r1 __reduce__KrBz.UnpicklableObject.__reduce__N)rRrSrTrr.r.r.r1UnpicklableObjectJs rzthe task to sendmatchrEruc3s|] }ttVqdSr>r#idrrr.r1riOz)test_parallel_pickling..rN)objectrrr"rmr.r.rr1test_parallel_picklingEs""rcCs0ttd|ddddtdDdksJdS)NrErrdr<timeoutcsr)gMbP?Nrrr.r.r1riUrjz0test_parallel_timeout_success..)rr"rmr;r.r.r1test_parallel_timeout_successRs  rcCsNtttd|ddddtdDWddS1s wYdS)NrErrcsr)rNrrr.r.r1ri^rjz-test_parallel_timeout_fail..r)rrr"rmr;r.r.r1test_parallel_timeout_failYs  "rc CsJtdur tttd|dddtddDWdn1s$wYtttd|ddddDWdn1sEwYtd|d}t|jdusZJt|j}tt|ddtddDWdn1szwYt|jdusJt|j|usJd dtd D|d d td DksJt|j}tt|d ddDWdn1swYt|jdusJt|j|usJddtd D|dd td DksJWdn1swYt|jdus Jn!tt tdddddDWdn 1s'wYtttddddtddDWdn 1sKwYtt tddddd tdDWdn 1sowYztdddd tddDWtdt y}zt |t rJWYd}~dSd}~ww)NrErccSg|] \}}tt||qSr.r#rAr/r?r@r.r.r1r2jz&test_error_capture..)rr)rrcSg|]}tt|qSr.r#r\rr.r.r1r2nrcSrr.rrr.r.r1r2vrcSrrrrr.r.r1r2rrcsrrrrr.r.r1rirz%test_error_capture..cSrr.rrr.r.r1r2rcSrrrrr.r.r1r2rcsrrrrr.r.r1rirrucSrr.rrr.r.r1r2rcSrr.rrr.r.r1r2rr)rdr}csr)T)rXNr#rYrgr.r.r1risrcss"|] \}}tt||VqdSr>rrr.r.r1ri z'The excepted error has not been raised.)r(rZeroDivisionErrorr"zipr-r=rrmr[rH Exception isinstancer,rW)r<r original_workersexr.r.r1test_error_capturebs~            !      rcCs|d|dS)Nz Consumed %s)append)queueitemr.r.r1consumerrrrzbatch_size, expected_queue) Produced 0 Consumed 0 Produced 1 Consumed 1 Produced 2 Consumed 2 Produced 3 Consumed 3 Produced 4 Consumed 4 Produced 5 Consumed 5r) rrrrrrrrrrrrcsTtfdd}td||dfdd|D|ks Jtdks(JdS) zC Test that with only one job, Parallel does act as a iterator. c3(tdD] }d||VqdSNz Produced %irmrrhrr.r1producer  z'test_dispatch_one_job..producerrrd batch_sizer<c3s|] }tt|VqdSr>r#rrrr.r1rirz(test_dispatch_one_job.. N)listr"r)r<r expected_queuer r.rr1test_dispatch_one_jobs  rcst}|fdd}tddd|dfdd|Dt}|d d ks,J|d d d }|dks;J|d}||ksFJtdksNJd S)zW Check that using pre_dispatch Parallel does indeed dispatch items lazily. c3rrrrrr.r1r r z/test_dispatch_multiprocessing..producerrErrk)rdr  pre_dispatchr<c3|] }ttdVqdS)rNr rrr.r1rirz0test_dispatch_multiprocessing..rrNrz Consumed anyrtrr)r(Managerrr"indexr)r<managerr queue_contentsfirst_consumption_indexproduced_3_indexr.rr1test_dispatch_multiprocessings    rcCs\tdddd}|ddtdD|jdksJWddS1s'wYdS) NrEautor5r csrer>rrgr.r.r1riz/test_batching_auto_threading..rr"rmrcompute_batch_size)rr.r.r1test_batching_auto_threadings"r cCs\tdd|d}|ddtdD|jdksJWddS1s'wYdS)NrErr csrer>rrgr.r.r1rirz2test_batching_auto_subprocesses..rrr)r<rr.r.r1test_batching_auto_subprocessess"r!cCsNtttddddddtdDWddS1s wYdS) zCMake sure that exception raised during dispatch are indeed capturedrEr)rdrr}csrer>rrgr.r.r1rirjz*test_exception_dispatch..rN)rrWr"rmr.r.r.r1test_exception_dispatchs  "r#cC tddddtdDdS)NrErucsrer>rr/jr.r.r1ri rjz(nested_function_inner..rrlrr.r.r1nested_function_inner  r'cCr$)NrErucsrer>)r#r'r%r.r.r1rirjz(nested_function_outer..rrlrr.r.r1nested_function_outerr(r)z'https://github.com/joblib/loky/pull/255)reasoncCstt}td|dddtdDWdn1swYt|j|j|j}d|}d|vs7Jd |vs=Jd |vsCJt|jtusLJdS) zEnsure errors for nested joblib cases gets propagated We rely on the Python 3 built-in __cause__ system that already report this kind of information to the user. rErccsrer>)r#r)rgr.r.r1rirjz1test_nested_exception_dispatch..rNr)r'rY) rrWr"rmrr`rtbr)r<excinfo report_linesreportr.r.r1test_nested_exception_dispatchs      r0c@s$eZdZdZdddZd ddZdS) FakeParallelBackendz8Pretends to run concurrently while running sequentially.rNcKs|||_||_|Sr>)r&rdr )rJrdr backend_argsr.r.r1 configure/s zFakeParallelBackend.configurecCs"|dkrttd|d}|S)Nrr)maxr(r'rJrdr.r.r1r&4sz$FakeParallelBackend.effective_n_jobsr)r)rRrSrTrUr3r&r.r.r.r1r1,s r1cCs8tttddWddS1swYdS)Nz unit-testingr;rrWr"r.r.r.r1test_invalid_backend:s  "r7cCsNtt}td|dWdn1swYdt|jvs%JdS)Nrrcz&n_jobs == 0 in Parallel has no meaning)rrWr"_initialize_backendrr)r<r-r.r.r1test_invalid_njobs?s r9cCs<ztdtdtvs JtdtksJWtd=dStd=w)N test_backend)r$r1r)r.r.r.r1test_register_parallel_backendFs   r;cCsTttksJztdtdddttksJWtt_ntt_wttks(JdS)Nr5T) make_default)rbDefaultBackendr$r)rr*r r.r.r.r1test_overwrite_default_backendOs r>cCs^t|ddt\}}|dksJtddksJt}|jdks%J|dkr;t|tks1Jt|jtks:JnB|dkrQt|t ksGJt|jt ksPJn4|dkrgt|t ks]Jt|jt ksfJn.| drt|t kstJt|jt ksJWddSWddSWddSWddSWddS1swYdS)Nrkrur3r4r5test_) r%r rar&r"rdr`rrr r startswithr1) backend_nameactive_backend active_n_jobsrr.r.r1check_backend_context_manager[s8    "rDcCsg|]}d|qS)ztest_backend_%dr.rgr.r.r1r2rrrkcCsf|tvr |t|tttksJt|ttksJtdddddtDttks1JdS)NrEr5rccss |] }|stt|VqdSr>)r#rD)r/rLr.r.r1ris   z/test_backend_context_manager..)r)setitemr1rbr=rDr" all_backends_for_context_manager)rr<r.r.r1test_backend_context_managervsrGc@seZdZdZdddZdS)ParameterizedParallelBackendz9Pretends to run conncurrently while running sequentially.NcCs|durtd||_dS)Nzparam should not be None)rWparam)rJrIr.r.r1rOs z%ParameterizedParallelBackend.__init__r>rQr.r.r.r1rHsrHcCs|tdtttksJtdddd?t\}}t|tks$J|j dks+J|dks1Jt }|j dks;J|j |usBJ|ddt dD}Wdn1sWwY|dd t dDksiJttkspJdS) N param_backend*rk)rIrdcsrer>r#rrgr.r.r1rirz=test_parameterized_backend_context_manager..rcSrr.rrgr.r.r1r2rz>test_parameterized_backend_context_manager..)rEr)rHrbr=r%r rar`rIr"rdrrm)rrBrCrrr.r.r1*test_parameterized_backend_context_managers    rMcCsttksJttdddd?t\}}t|tksJ|jdks&J|dks,Jt}|j dks6J|j |us=J|ddt dD}Wdn1sRwY|ddt dDksdJttkskJdS) N+)rIrrucsrer>rLrgr.r.r1rirzFtest_directly_parameterized_backend_context_manager..cSrr.rrgr.r.r1r2rzGtest_directly_parameterized_backend_context_manager..) rbr=r%rHr rar`rIr"rdrrm)rBrCrrr.r.r13test_directly_parameterized_backend_context_managers   rOcCstdtS)Nr)rosgetpidr.r.r.r1sleep_and_return_pidsrRcCs<ttksJtdksJtddddtdDS)NrrErucs|]}ttVqdSr>)r#rRrr.r.r1risz"get_nested_pids..)rbrr"_effective_n_jobsrmr.r.r.r1get_nested_pidss rUcs eZdZdZfddZZS) MyBackendz:Backend to test backward compatibility with older backendscstt|dSr_)superrVget_nested_backendr __class__r.r1rXszMyBackend.get_nested_backend)rRrSrTrUrX __classcell__r.r.rYr1rVsrVback_compat_backend)r5r4r3r\cCsht|&tddddtdD}|D] }tt|dks!JqWddS1s-wYdS)NrErucsrSr>)r#rUrr.r.r1ris  z6test_nested_backend_context_manager..rr)r%r"rmrset)r< pid_groups pid_groupr.r.r1#test_nested_backend_context_managers "r`)rErtNcs~ddtddfddtdDtdtddfddtdDWddS1s8wYdS) NcSs0tt|ks Jt|}t|ksJdSr>)rbr)r&r"rT)expected_backend_typeexpected_n_jobr.r.r1check_nested_backendsz?test_nested_backend_in_sequential..check_nested_backendrruc3s|] }tddVqdS)r4rNr#r)rcr.r1ri  z4test_nested_backend_in_sequential..rc3s|] }tVqdSr>rdrr<rcrdr.r1rire)r"rmr%)r<rdr.rfr1!test_nested_backend_in_sequentials  "rgcCs@t|\}}|j|ksJWddS1swYdSr>)r% nesting_level) inner_backendexpected_levelr<rdr.r.r1check_nesting_levels"rk outer_backendricszt|dtd|dfddtdDtddtfddtdDWddS1s6wYdS) NrrErcc3rrr#rkrrir.r1rirez-test_backend_nesting_level..rruc3rrrmrrnr.r1ris)rkr"rmr%)rlrir.rnr1test_backend_nesting_level s  "rocsddlGfdddt}td|ddtd"\}}tddfd d td D|jd ks5JWddS1s@wYdS) NrcseZdZdZjddZdS)z)test_retrieval_context..MyBackendrcss|jd7_dVdSrvrrr.r.r1retrieval_context!s z;test_retrieval_context..MyBackend.retrieval_contextN)rRrSrTrhcontextmanagerrpr.) contextlibr.r1rVsrV retrievalcSstddddt|DS)NrErucsrer>rrgr.r.r1ri)rz>test_retrieval_context..nested_call..rl)nr.r.r1 nested_call(sz+test_retrieval_context..nested_callrEruc3s|] }t|VqdSr>rdrg)rur.r1ri,s  z)test_retrieval_context..rr)rrrr$r%r"rmrh)rVbarr.)rrrur1test_retrieval_contexts "rwcCstd}t|t|dS)Nfoobar)r,reprpickledumps)er.r.r1test_joblib_exception5sr}cCsxtt}tt |ddWdn1swYtt}tt |dWddS1s5wYdS)Nrrr?)r!rArrr\r-) safe_divisionsafe_interruptr.r.r1test_safe_function>s    "rr )rrtgQ?cCs8ttt|dWddS1swYdS)Nr r6rr.r.r1test_invalid_batch_sizeHs  "rz)n_tasks, n_jobs, pre_dispatch, batch_size) )rErErr)rErErdrrrErdr)irErdrr)rrrdr)rrdr)r 2 * n_jobsr)rrr)rrrrV)rrrrcCsN|||d}ddt|D}tdi|ddt|D}||ks%JdS)N)rdrr cSrr.rrgr.r.r1r2`rz0test_dispatch_race_condition..csrer>rfrgr.r.r1riarz/test_dispatch_race_condition..r.r)n_tasksrdrr paramsrrr.r.r1test_dispatch_race_conditionNs  rcCs8t}tddd}|jd}|}||ksJdS)NrEr3rccontext)r(get_start_methodr" _backend_argsget)mp_start_methodrr start_methodr.r.r1test_default_mp_contextes   rcs^|dkr td}tjd}|ddtjtd|dfddt dDdS) Nr3r7rKirErcc3s"|] }ttjjVqdSr>)r#rdotTrgrKr.r1rirzAtest_no_blas_crash_or_freeze_with_subprocesses..) r(r6rrandom RandomStaterandnrrr"rm)r<rngr.rr1.test_no_blas_crash_or_freeze_with_subprocessesns    rafrom joblib import Parallel, delayed def square(x): return x ** 2 backend = "{}" if backend == "spawn": from multiprocessing import get_context backend = get_context(backend) print(Parallel(n_jobs=2, backend=backend)( delayed(square)(i) for i in range(5))) cCsB|dkrtdkrtdt|}ttjd|gddddS)Nr3forkzVRequire fork start method to use interactively defined functions with multiprocessing.z-cr\[0, 1, 4, 9, 16\]r stdout_regex) r(rpytestskip,UNPICKLABLE_CALLABLE_SCRIPT_TEMPLATE_NO_MAINformatrsys executable)r<coder.r.r12test_parallel_with_interactively_defined_functionss    raimport sys # Make sure that joblib is importable in the subprocess launching this # script. This is needed in case we run the tests from the joblib root # folder without having installed joblib sys.path.insert(0, {joblib_root_folder!r}) from joblib import Parallel, delayed def run(f, x): return f(x) {define_func} if __name__ == "__main__": backend = "{backend}" if backend == "spawn": from multiprocessing import get_context backend = get_context(backend) callable_position = "{callable_position}" if callable_position == "delayed": print(Parallel(n_jobs=2, backend=backend)( delayed(square)(i) for i in range(5))) elif callable_position == "args": print(Parallel(n_jobs=2, backend=backend)( delayed(run)(square, i) for i in range(5))) else: print(Parallel(n_jobs=2, backend=backend)( delayed(run)(f=square, x=i) for i in range(5))) z!def square(x): return x ** 2 zcdef gen_square(): def square(x): return x ** 2 return square square = gen_square() zsquare = lambda x: x ** 2 define_funccallable_position)r#rkwargsc Csv|dvr|tks tjdkrtdtj|||tj tj t j d}| d}| |ttj|jgddddS) N)r3r7win32zNot picklable with pickle)rr<rjoblib_root_folderzunpicklable_func_script.pyrrr) SQUARE_MAINrplatformrr)UNPICKLABLE_CALLABLE_SCRIPT_TEMPLATE_MAINrrPpathdirnamejoblib__file__rwriterrstrpath)r<rrtmpdirr code_filer.r.r10test_parallel_with_unpicklable_functions_in_argss     raimport sys # Make sure that joblib is importable in the subprocess launching this # script. This is needed in case we run the tests from the joblib root # folder without having installed joblib sys.path.insert(0, {joblib_root_folder!r}) from joblib import Parallel, delayed from functools import partial class MyClass: '''Class defined in the __main__ namespace''' def __init__(self, value): self.value = value def square(x, ignored=None, ignored2=None): '''Function defined in the __main__ namespace''' return x.value ** 2 square2 = partial(square, ignored2='something') # Here, we do not need the `if __name__ == "__main__":` safeguard when # using the default `loky` backend (even on Windows). # The following baroque function call is meant to check that joblib # introspection rightfully uses cloudpickle instead of the (faster) pickle # module of the standard library when necessary. In particular cloudpickle is # necessary for functions and instances of classes interactively defined in the # __main__ module. print(Parallel(n_jobs=2)( delayed(square2)(MyClass(i), ignored=[dict(a=MyClass(1))]) for i in range(5) )) )rcCs.|d}|tttj|jgddddS)Nz(joblib_interactively_defined_function.pyrr)rr)rr5INTERACTIVE_DEFINED_FUNCTION_AND_CLASS_SCRIPT_CONTENTrrrrrscriptr.r.r1Btest_parallel_with_interactively_defined_functions_default_backends    rauimport sys # Make sure that joblib is importable in the subprocess launching this # script. This is needed in case we run the tests from the joblib root # folder without having installed joblib sys.path.insert(0, {joblib_root_folder!r}) from joblib import Parallel, delayed, hash import multiprocessing as mp mp.util.log_to_stderr(5) class MyList(list): '''MyList is interactively defined by MyList.append is a built-in''' def __hash__(self): # XXX: workaround limitation in cloudpickle return hash(self).__hash__() l = MyList() print(Parallel(n_jobs=2)( delayed(l.append)(i) for i in range(3) )) cCs0|d}|tttj|jgdddddS)Nz)joblib_interactive_bound_method_script.pyz\[None, None, None\] LokyProcess)r stderr_regexr)rr9INTERACTIVELY_DEFINED_SUBCLASS_WITH_METHOD_SCRIPT_CONTENTrrrrrr.r.r15test_parallel_with_interactively_defined_bound_method9s    rcCs"tg}tdd|gksJdS)NrEru)iterr")exhausted_iteratorr.r.r1%test_parallel_with_exhausted_iteratorCsrcCs"t|tjs tdt||S)Nz#Expected np.memmap instance, got %r)rrmemmap TypeErrorr`copyrr.r.r1 check_memmapHs rcCsdd}tdd|ddd|dD}t||t|D] \}}tj||qtd d|dd d|dD}t||t|D] \}}tj||qBdS) Ncss*t|D] }tjdtjd|VqdS)Nrdtype)rmronesfloat32)rtrhr.r.r1generate_arraysWs zBtest_auto_memmap_on_arrays_from_generator..generate_arraysrErrd max_nbytesr<csrer>r#rr/rKr.r.r1ri\rjz.rrcsrer>rrr.r.r1ridrj)r"rrrtestingassert_array_equal)r<rrresultrr.r.r1)test_auto_memmap_on_arrays_from_generatorOsrcCs|Sr>r.)argr.r.r1identityjrPrcs|dj}tj}tj|ddtj|ddg}t||t|ddt ddfdd d D\}t d tj s;Jd j |ksDJtj ||dS) Nz test.mmapuint8rr) mmap_moderEruc3rr>)r#rrrr.r1rivrz.test_memmap_with_big_offset..)rr)rrmmapALLOCATIONGRANULARITYrzerosrr r r"rroffsetrr)rfnamesizeobjrr.rr1test_memmap_with_big_offsetns   rcCstd}tddddtdDWdn1swYt|dks*J|d}t|jts6Jt|jdks?JdS)Nr)rcsrer>rfrgr.r.r1ri~rzFtest_warning_about_timeout_not_supported_by_backend..2rzThe backend class 'SequentialBackend' does not support timeout. You have set 'timeout=1' in Parallel but the 'timeout' parameter will not be used.)rr"rmrrr UserWarningr)rrr.r.r13test_warning_about_timeout_not_supported_by_backend|s  r)rrEr|rtcCsvdgdgd}ttt}t||ddd|DWdn1s(wYt|}|dks9JdS)NrKrrrccss|] }ttj|VqdSr>r#rZrrgr.r.r1rirz%test_abort_backend..)rrrZr")rdr<delayst_startdtr.r.r1test_abort_backends   rcCs|j}tdd||d#}|ddtjdgdDtt|dks'JWdn1s1wYtdD] }t|sCn td q:t d tdd|d }|d dtjdgdDtdD]}t|sqdStd qgt d ) NrEr)rdrr< temp_foldercsrer>rrr.r.r1rirz(test_memmapping_leaks..rrrrz/temporary directory of Parallel was not removedrcsrer>rrr.r.r1rir) rr"rrrrPlistdirrmrAssertionError)r<rrrr.r.r1test_memmapping_leakss(       r)Nr4r5cCs<td|dddtdD}|ddtdDksJdS)NrErccss |] }tdd|VqdS)cSrCrDr.rFr.r.r1sz2test_lambda_expression...Nrdrgr.r.r1ris z)test_lambda_expression..rcSsg|]}|dqS)rEr.rgr.r.r1r2rz*test_lambda_expression..rl)r<rr.r.r1test_lambda_expressionsrcsd}d}d|td||d}|fddt|D|jj|jjks&J|jj|jjks0J|fddt|D|jj|jjksGJ|jj|jjksQJd S) zCTest that a parallel backend correctly resets its batch statistics.rEig@r)r}rdr<c3|] }ttjVqdSr>rrg task_timer.r1rirz6test_backend_batch_statistics_reset..c3rr>rrgrr.r1rirN)r"rmr_effective_batch_size_DEFAULT_EFFECTIVE_BATCH_SIZE_smoothed_batch_duration _DEFAULT_SMOOTHED_BATCH_DURATION)r<rdn_inputsrr.rr1#test_backend_batch_statistics_resets$ rcCsdD];}tt|djtksJt|dd}t|jtksJt|dd}t|jtks.Jt|dd}t|jtks=Jqtdd dd }t|jtksNJtd ddtdd }t|jtkscJ|jdksjJWdn1stwYtd ddtd dd}t|jtksJ|jd ksJWdn1swYtd ddtdd }t|jtksJ|jdksJWdn1swYtd ddtd dd}t|jtksJ|jd ksJWddS1swYdS)N)rrErtruthreadsrdprefer processes sharedmemrdrequirerEr4)rdr<rrrkrr)r`r"rr rr%rd)rdrr.r.r1$test_backend_hinting_and_constraintss<       "rcCszGdddt}t|&tddd}t|j|ksJtddd}t|j|ks,JWdn1s6wYGdd d t}t|Otddd}t|j|ksXJ|\}}|d ksdJ|d ksjJtddd d }t|jtkszJ|\}}d }||ksJ|d ksJWdn1swYtt t|ddWddS1swYdS)Nc@$eZdZdZdZddZddZdS)z[test_backend_hinting_and_constraints_with_custom_backends..MyCustomThreadingBackendTcSrIr>r.rr.r.r1 apply_async rPzgtest_backend_hinting_and_constraints_with_custom_backends..MyCustomThreadingBackend.apply_asynccS|Sr>r.r5r.r.r1r&rPzltest_backend_hinting_and_constraints_with_custom_backends..MyCustomThreadingBackend.effective_n_jobsNrRrSrTsupports_sharedmem use_threadsrr&r.r.r.r1MyCustomThreadingBackend   r rErrrrc@r)z\test_backend_hinting_and_constraints_with_custom_backends..MyCustomProcessingBackendFcSrIr>r.rr.r.r1rrPzhtest_backend_hinting_and_constraints_with_custom_backends..MyCustomProcessingBackend.apply_asynccSrr>r.r5r.r.r1r&!rPzmtest_backend_hinting_and_constraints_with_custom_backends..MyCustomProcessingBackend.effective_n_jobsNrr.r.r.r1MyCustomProcessingBackendr r r+r)rdrr}zUsing ThreadingBackend as joblib.Parallel backend instead of MyCustomProcessingBackend as the latter does not provide shared memory semantics.r<r) rr%r"r`r readouterrrstriprrW)capsysr rr outerrrr.r.r19test_backend_hinting_and_constraints_with_custom_backendss0        "rcCstt tddWdn1swYtt tddWdn1s-wYtttdddWdn1sGwYtttdddWdn1sawYtttd ddWddS1s|wYdS) Ninvalidrrrr)rrr4r r3r6r.r.r.r1,test_invalid_backend_hinting_and_constraints:s       "rcCsFtj}td}ttjtsJ|ttjt|us!JdS)Nr5)r"rr%rr unregisterr`)defaultpbr.r.r1test_global_parallel_backendNs rcCsPdd}|td<tdttjtsJWddS1s!wYdS)NcSs ttd<dS)Nfoo)rr)r.r.r.r1 register_fooY z,test_external_backends..register_foor)r+r%rr"rr)rr.r.r1test_external_backendsXs  "rc stdd4}t|jj|jjfg}dkr|WdS|fddtdD}||dWdS1s.r)r"r`rrRrhrm)rrr this_levelrr.r r1rbs  $rcCst|dd t}Wdn1swYtdkr#d}d}nd}d}|d}|df|dfd|fd|fg}||ksAJdS)NrErurrrBackendr)r%rr'title)r<backend_types_and_levelssecond_level_backend_type max_leveltop_level_backend_typeexpected_types_and_levelsr.r.r1test_nested_parallelism_limitns  r)zThis test requires daskc Cstjddd}tjtdtjd}tdD].}td t|d}Wdn1s+wYt |dks8Jt dd |DsCJqtd t}Wdn1sVwYt |dkscJt d d |DsnJdS) NrE) n_workersthreads_per_workergcArdask)datarcs|] \}}|dkVqdSDaskDistributedBackendNr.r/rrr.r.r1riz4test_nested_parallelism_with_dask..csr.r/r.r1r.r.r1rir2) distributedClientrrintrrmr%rrr)clientr-rhr$r.r.r1!test_nested_parallelism_with_dasks"    r7cCstddtdDS)z6A horrible function that does recursive parallel callscsrSr>)r#_recursive_parallelrgr.r.r1risz&_recursive_parallel..rErl) nesting_limitr.r.r1r8sr8c Cst|ddtt }tWdn1swYWdn1s(wY|j}|dkr@t|tr@tddSt|t sGJdS)NrErur4z1Loky worker crash when serializing RecursionError) r%r BaseExceptionr8rrrrxfailRecursionError)r<r-excr.r.r1test_thread_bomb_mitigations r>cCs*i}dD] }tj|||<q|tdfS)N)OMP_NUM_THREADSOPENBLAS_NUM_THREADSMKL_NUM_THREADSVECLIB_MAXIMUM_THREADSNUMEXPR_NUM_THREADSNUMBA_NUM_THREADS ENABLE_IPCr)rPenvironrr)env_varsvarr.r.r1_run_parallel_sums rIzNeed OpenMP helper compiledcCstd|dddtdD}ttdd}|D]-\}}||ks#J|D]\}}|dr9|t|ks8Jq'|dks?J|dksEJq'qdS) NrErccsrSr>)r#rIrr.r.r1ri  z-test_parallel_thread_limit..r_THREADSrE1)r"rmr4r'itemsendswithr)r<rexpected_num_threadsworker_env_varsomp_num_threadsrrr.r.r1test_parallel_thread_limits    rRz%This test requires dask NOT installedcCs:ttdd tdWddS1swYdS)NzPlease install daskrr,)rrWr%r.r.r.r1)test_dask_backend_when_dask_not_installeds "rSc CsGdddt}d}t|6tjt|dtddddtdDWdn1s0wYWddSWddS1sHwYdS) Nc@s&eZdZddZdddZddZdS) z3test_zero_worker_backend..ZeroWorkerBackendc_dSr_r.)rJrrr.r.r1r3rPz=test_zero_worker_backend..ZeroWorkerBackend.configureNcSr)NzNo worker availabler)rJfunccallbackr.r.r1rrBz?test_zero_worker_backend..ZeroWorkerBackend.apply_asynccSrTr_r.r5r.r.r1r&rPzDtest_zero_worker_backend..ZeroWorkerBackend.effective_n_jobsr>)rRrSrTr3rr&r.r.r.r1ZeroWorkerBackends  rWz&ZeroWorkerBackend has no active workerrrErucsrer>rrgr.r.r1rirz+test_zero_worker_backend..)rr%rrrr"rm)rW expected_msgr.r.r1test_zero_worker_backends "rYcsdadddks JtddfddtdD}t|dhks&Jdadks/Jtddfd dtdD}t|dhksHJdS) Nzoriginal valuecSstSr>)MY_GLOBAL_VARIABLEr.r.r.r1 check_globalssz@test_globals_update_at_each_parallel_call..check_globalsrEruc3|]}tVqdSr>rdrgr[r.r1rirJz.z changed valuec3r\r>rdrgr]r.r1rirJ)rZr"rmr])workers_global_variabler.r]r1)test_globals_update_at_each_parallel_callsr_cCs4ddl}|jdd}|||ddlm}|S)Nrr)threadpool_info)numpyrrr threadpoolctlr`)rrKr`r.r.r1_check_numpy_threadpool_limits s   rccCs4|D]}|d|dkr|dSqtd|)Nfilepath num_threadsz,An unexpected module was loaded in child: {})rWr) child_module parent_info parent_moduler.r.r1_parent_max_num_threads_fors ricCs@|D]}|D]}t||}t|||h}|d|vsJqqdS)Nre)rimin) workers_inforgrechild_threadpool_inforfparent_max_num_threadsrr.r.r1check_child_num_threadssrn)rErr|rtcCsbt}t|dkrtjddt|dddtdD}t|}tt|d}t |||dS) Nr&Need a version of numpy linked to BLASmsgrucsrSr>r#rcrgr.r.r1ri4rJz6test_threadpool_limitation_in_child..rEr) rcrrrr"rmr&r4r'rn)rdrgworkers_threadpool_infosexpected_child_num_threadsr.r.r1#test_threadpool_limitation_in_child(s  ruinner_max_num_threads)rrErNrEcCst}t|dkrtjddtd|dt|dddtd D}Wdn1s.wYt|}|durDtt |d }n|}t |||dS) Nrrorpr4rvrucsrSr>rrrgr.r.r1riLrJz>test_threadpool_limitation_in_child_context..rEr) rcrrrr%r"rmr&r4r'rn)rdrvrgrsrtr.r.r1+test_threadpool_limitation_in_child_context>s   rxvar_name)r@rAr?c stddddtj}z_dtj<t|dfddtd D}|ddgks0Jtd d d t|dfd dtd D}Wdn1sRwY|ddgks_JW|durjtj=dS|tj<dS|durztj=w|tj<w)NTreusecSs tj|Sr>)rPrFr)ryr.r.r1_get_envfrz>test_threadpool_limitation_in_child_override.._get_env4ruc3|] }tVqdSr>rdrgr|ryr.r1rimrjz?test_threadpool_limitation_in_child_override..rEr4rrwc3r~r>rdrgrr.r1rirrjrL)rshutdownrPrFrr"rmr%)rdryoriginal_var_valuerr.rr1,test_threadpool_limitation_in_child_overrideYs*     rcCs>ttddt|ddWddS1swYdS)Nz#does not acc.*inner_max_num_threadsrrrw)rrr%r;r.r.r11test_threadpool_limitation_in_child_context_error}s"r)rErrtcCsJdd}||tdd}tdD]}||tdd}||ks"JqdS)Ncs,tdt|dfddtdDdS)Nrruc3rr>)r#sumrgrFr.r1rirzAtest_loky_reuse_workers..parallel_call..rrur.rFr1 parallel_calls$z.test_loky_reuse_workers..parallel_callTrzr)rrm)rdrfirst_executorrexecutorr.r.r1test_loky_reuse_workerss   r)F)rr)rkr>)rUrPrrZrr5 tracebackrmathrrrzrr3rr importlibr rr r r joblib.externals.lokyrjoblib.test.commonrrrjoblib.testingrrrrrr&joblib.externals.loky.process_executorrrrposix ImportError _openmp_test_helper.parallel_sumrr3joblib._parallel_backendsrrrrr r!joblib.parallelr"r#r$r%r&r'r(r)r*r+joblib.my_exceptionsr,r-sortedkeysALL_VALID_BACKENDSPROCESS_BACKENDSPARALLEL_BACKENDShasattrrr6r=r=rArGrrHrYr\r^rbrorqrsmarkr{rrrrrrrrrrrrrrrrrrrr r!r#r'r)r;r0r1r7r9r;r>rDrFextendrmrGrHrMrOrRrU_parallel_backendsrVr`rgrkrorwr}rrrrrrrrr SQUARE_LOCAL SQUARE_LAMBDArrrrrrrrrrrrrrrrrrrrrrrrrr)r7r8r>rIrRrSrYr_rcrirnrurxrrrr.r.r.r1s0                                             I                         $ (           #  .4           !