!gm' dZddlmZddlZdZdZddZGddZe dk(rddl m Z d Z ejje Zd gZd evreeeeed eeed ej&ej)ej+ZeeeZe j0eee j0eeeed zej2ej4eej4eed Zej8eDcgc]}ej;|d c}Ze j>e j0eeeeZ e j>e j0ee e j>e jBeddejDe ejDez ej&ej)ej+dZ#e j>e jBe#ddejDee#ejDe#z ej4eZ$e$dde dz Z%e j>e jBe%ddejDee%ejDe%z eeZ&ee&jOee&jQe&jRee&jUgdee&jQgdej&ej)ej+dZej2ej4eej4eed ZeeZ e j>e j0ee ej2e edZ+e+e Z,e j0e,e ejZej4eeej4eddZ.e.e Z/e j0e/e ededejDe,j)edejDe j)yycc}w)a from David Huard's scipy sandbox, also attached to a ticket and in the matplotlib-user mailinglist (links ???) Notes ===== out of bounds interpolation raises exception and would not be completely defined :: >>> scoreatpercentile(x, [0,25,50,100]) Traceback (most recent call last): ... raise ValueError("A value in x_new is below the interpolation " ValueError: A value in x_new is below the interpolation range. >>> percentileofscore(x, [-50, 50]) Traceback (most recent call last): ... raise ValueError("A value in x_new is below the interpolation " ValueError: A value in x_new is below the interpolation range. idea ==== histogram and empirical interpolated distribution ------------------------------------------------- dual constructor * empirical cdf : cdf on all observations through linear interpolation * binned cdf : based on histogram both should work essentially the same, although pdf of empirical has many spikes, fluctuates a lot - alternative: binning based on interpolated cdf : example in script * ppf: quantileatscore based on interpolated cdf * rvs : generic from ppf * stats, expectation ? how does integration wrt cdf work - theory? Problems * limits, lower and upper bound of support does not work or is undefined with empirical cdf and interpolation * extending bounds ? matlab has pareto tails for empirical distribution, breaks linearity empirical distribution with higher order interpolation ------------------------------------------------------ * should work easily enough with interpolating splines * not piecewise linear * can use pareto (or other) tails * ppf how do I get the inverse function of a higher order spline? Chuck: resample and fit spline to inverse function this will have an approximation error in the inverse function * -> does not work: higher order spline does not preserve monotonicity see mailing list for response to my question * pmf from derivative available in spline -> forget this and use kernel density estimator instead bootstrap/empirical distribution: --------------------------------- discrete distribution on real line given observations what's defined? * cdf : step function * pmf : points with equal weight 1/nobs * rvs : resampling * ppf : quantileatscore on sample? * moments : from data ? * expectation ? sum_{all observations x} [func(x) * pmf(x)] * similar for discrete distribution on real line * References : ? * what's the point? most of it is trivial, just for the record ? Created on Monday, May 03, 2010, 11:47:03 AM Author: josef-pktd, parts based on David Huard License: BSD Nctj|}t|}tjtj |tj |}||dz S)zReturn the score at the given percentile of the data. Example: >>> data = randn(100) >>> scoreatpercentile(data, 50) will return the median of sample `data`. Y@)nparray empiricalcdf interpolateinterp1dsort)data percentilepercdf interpolators c/var/www/dash_apps/app1/venv/lib/python3.12/site-packages/statsmodels/sandbox/stats/stats_dhuard.pyscoreatpercentilerVsM ((: C t C'' bggdmDL D !!ct|}tjtj|tj|}||dzS)aDReturn the percentile-position of score relative to data. score: Array of scores at which the percentile is computed. Return percentiles (0-100). Example r = randn(50) x = linspace(-2,2,100) percentileofscore(r,x) Raise an error if the score is outside the range of data. r)rrr rr )r scorerrs rpercentileofscorerds@ t C'' rwws|DL  t ##rcjtjtj|dz}t|}|j}|dk(r |dz |z }|S|dk(r ||dzz }|S|dk(r |dz |z }|S|dk(r |dz |dzz }|S|d k(r |dz |d zz }|S|d k(r |d z |d zz }|St d)aReturn the empirical cdf. Methods available: Hazen: (i-0.5)/N Weibull: i/(N+1) Chegodayev: (i-.3)/(N+.4) Cunnane: (i-.4)/(N+.2) Gringorten: (i-.44)/(N+.12) California: (i-1)/N Where i goes from 1 to N. ?hazen?weibull california chegodayev333333?皙?cunnane皙? gringorten)\(?Q?[Unknown method. Choose among Weibull, Hazen,Chegodayev, Cunnane, Gringorten and California.)rargsortlenlower ValueError)r methodiNrs rrrvs  2::d#$r)A D A \\^F uai J 9 2h J < tQh J < tadm J 9 tadm J < uquo JKL Lrc2eZdZdZdZddZdZdZd dZy) HistDistzDistribution with piecewise linear cdf, pdf is step function can be created from empiricial distribution or from a histogram (not done yet) work in progress, not finished cRtj||_tj|jj |jj g|_tj|}|||_tj||_ |j}tj||_ tj|j|j|_tj|j|j|_y)N)r atleast_1dr rminmaxbinlimitr% _datasortedrankingrr _empcdfsortedrr cdfintpppfintp)selfr sortindrs r__init__zHistDist.__init__sMM$' $))--/499==?!CD **T"=zz'* !WWS\"++D,<,P>PQ "++D,>,>@P@PQ rNc||j}|j}n+tjtj|dz}t |}|j }|dk(r |dz |z }|S|dk(r ||dzz }|S|dk(r |dz |z }|S|dk(r |dz |dzz }|S|d k(r |dz |d zz }|S|d k(r |d z |d zz }|St d)aAReturn the empirical cdf. Methods available: Hazen: (i-0.5)/N Weibull: i/(N+1) Chegodayev: (i-.3)/(N+.4) Cunnane: (i-.4)/(N+.2) Gringorten: (i-.44)/(N+.12) California: (i-1)/N Where i goes from 1 to N. rrrrrrrrrr r!r"r#r$)r r4rr%r&r'r()r8r r)r*r+rs rrzHistDist.empiricalcdfs  <99D A 2::d+,r1A I W S5!)C y QrT(C | #R4(C | #R4!B$-C y R4!B$-C | #S51S5/C  OP Prc$|j|Sz& this is score in dh )r6)r8rs rcdf_empzHistDist.cdf_emps ||E""rc$|j|Sr=)r7)r8quantiles rppf_empzHistDist.ppf_emps ||H%%rcTt|j}|dk(r/|jd|jdz }d|z|dzz}n-|dk(r(dtj|jz|dzz}tj |j z |_|jS)zFind the optimal number of bins and update the bin countaccordingly. Available methods : Freedman Scott Freedman??gUUUUUUտScottgQ @)r&r rArstdptpr2nbin)r8r)nobsIQRwidths roptimize_binningzHistDist.optimize_binnings 499~ : ,,t$t||D'99CsFD5M)E W_266$)),,te}rArNrrr-r-s" R&R#&rr-__main__drFr2)k)rErrD)ggпrrErigQ?)rYsznegative densityz(np.diff(ppfs)).min()z(np.diff(cdf_ongrid)).min())rO)0rSscipy.interpolaternumpyrrrrr-rPmatplotlib.pyplotpyplotpltrKrandomrandnxexamplesprintlinspacer0r1xsuppposplotInterpolatedUnivariateSpliner empr derivativespdfempfigure cdf_ongridstepdiffxsupp2xsoxshistdrNr>r2rAr7ppfsUnivariateSplineppfempppfe)xis0rr}sQd( "$$!H``H z# D AsHH}Q 3'( 2&' AEEGQUUW-5) "1c*CE2 5K 4 4WRWWQZ UV@X[\ ]EBb3??2.q1BC vZ   #  sGBGGJ/>?QUUWaeegr2 WRWWS[1'"''&/ABbggaj 47^ CRR)'"''"+56 QKE % "# %-- '( %--) *+ %--3 45 BKK# .E0 00GBGGLQRO