Statistics
| Branch: | Tag: | Revision:

root / kamaki / clients / pithos.py @ f5f2dc53

History | View | Annotate | Download (35.4 kB)

# Date Author Comment
f5f2dc53 02/21/2013 12:50 pm Stavros Sachtouris

Rename tests to livetest in kamaki.clients

Update documentation to reflect these changes

fce31e83 02/12/2013 03:41 pm Stavros Sachtouris

URL-encode requests

2005b18e 02/11/2013 05:42 pm Stavros Sachtouris

Fix some more pep8 issues, more later

24ff0a35 02/11/2013 05:19 pm Stavros Sachtouris

Fix pep8 compliance issues everywhere

ca092af4 01/25/2013 05:58 pm Stavros Sachtouris

Adjust up to store-overwrite

745d938b 01/21/2013 06:00 pm Stavros Sachtouris

Tide up _upload_missing_blocks code + progress chk

progress check in upload: don't let progress bar raise exceptions

7eda693f 01/21/2013 03:23 pm Stavros Sachtouris

Doc updates and minor improvements

3e7d1e0e 01/21/2013 01:53 pm Stavros Sachtouris

Upload progress bar advances only if block uploads

was: progress bar advances if thread is initialized. In case of thread
failures, progress bar was re-initialized for a next round of uploads

now: if a block is uploaded, progress bad advances. There is one upload bar...

81e34150 01/15/2013 02:55 pm Stavros Sachtouris

Merge branch 'feature-output' into develop

Conflicts:
kamaki/clients/__init__.py
kamaki/clients/pithos.py

16c895db 01/14/2013 04:22 pm Stavros Sachtouris

More retries, dynamic upload thread limit

Even the slightest progress causes upload to retry
if 502 client error is received, # of threads are limited to current thread

7644c38e 01/14/2013 03:39 pm Stavros Sachtouris

Handle thread failures in download

If some threads fail for some reason, download re-tries 2 more times,
provided there is some progress

706fc940 01/11/2013 02:05 pm Stavros Sachtouris

upload does not overwrite remote objs by default

Now you can do this to upload all you .jpg imgs to myImgDir directory object:

for img in *.jpg;
do
kamaki store upload $img mycontainer:myImgDir
done

93542587 12/20/2012 02:24 pm Stavros Sachtouris

Correct publish output, more details in storelist

fba9be97 12/18/2012 11:23 am Stavros Sachtouris

Merge remote-tracking branch 'origin/develop' into feature-windows

Conflicts:
kamaki/cli/__init__.py
kamaki/cli/argument.py
kamaki/cli/command_shell.py
kamaki/cli/command_tree.py
kamaki/cli/commands/__init__.py
kamaki/cli/commands/astakos_cli.py
kamaki/cli/commands/config_cli.py...

d56d6f9f 12/12/2012 09:00 pm Dionysis Zindros

Merge branch 'develop' into feature-windows

277ca4ed 12/12/2012 08:58 pm Dionysis Zindros

Prepare to merge into develop

9730e234 12/12/2012 08:38 pm Dionysis Zindros

Explicit error message for offset/size mismatch.

To assist porting to Windows, offset/size mismatch when reading files is now
being clearly reported with detailed error messages.

f91bc6b1 12/10/2012 04:08 pm Stavros Sachtouris

Add info in container-not-found error

319be41b 12/10/2012 01:50 pm Stavros Sachtouris

Fix typo (get_object_hashmapp)

5260a313 12/07/2012 07:24 pm Stavros Sachtouris

Pithos+ publish returns access url

6069b53b 12/07/2012 04:05 pm Stavros Sachtouris

Trace errors in cli.commands mechanism-not prpgtd

1d329d27 12/05/2012 07:24 pm Stavros Sachtouris

Better pithos upload progressbar

4c1882ab 12/04/2012 07:35 pm Stavros Sachtouris

add unittests requirment (progress)

d88ba587 12/04/2012 01:17 pm Stavros Sachtouris

Update docs and doc-parsed comments in storage

docs: replace Glance with Plankton
storage: standard sphincs-parsable comments

5fb18128 11/30/2012 06:29 pm Stavros Sachtouris

There are always some typos...

bc223d91 11/30/2012 06:24 pm Stavros Sachtouris

Update comment docs at pithos client

4375e020 11/30/2012 06:06 pm Stavros Sachtouris

Change order of params for get_object_meta

961e6040 11/28/2012 04:54 pm Stavros Sachtouris

Rearrange sections in docs, add developer corner

4067cdaf 11/26/2012 02:12 pm Stavros Sachtouris

Fix sphinx auto-doc syntax warning

2b74ab4a 11/19/2012 06:23 pm Stavros Sachtouris

Unittest parallel server creation

cccff590 11/16/2012 06:17 pm Stavros Sachtouris

Thread limit can be set by user

cad39033 11/16/2012 05:31 pm Stavros Sachtouris

Gradual thread inc/decrease in up/download

Thread inc/decrease is now generic for all clients that might need it
with the method _watch_thread_limit

Algorithm: set a small thread limit and when reached wait for active threads to
finish. Measure avg time. If avg time decreases, increase thread limit...

e9abe82b 11/16/2012 02:10 pm Stavros Sachtouris

Avoid to many threads in upload

22beceb3 11/05/2012 04:35 pm Stavros Sachtouris

Minor improvements in pithos unitesting

f27ed9a0 11/05/2012 03:55 pm Stavros Sachtouris

Sync progressbar to threads, adjust ctrl-c handler

In pithos download/upload ctrl-c is caught when multithreading,
and wait for active threads to finish before exiting

e02728f9 11/02/2012 07:28 pm Stavros Sachtouris

Use standard threading instead of gevent/greenlets

gevent depedency is not needed after that

3dabe5d2 11/02/2012 01:16 pm Stavros Sachtouris

Make all source pep8 compliant

68da89ac 10/31/2012 11:50 am Stavros Sachtouris

Merge branch 'master' of https://code.grnet.gr/git/kamaki

Conflicts:
kamaki/__init__.py
kamaki/cli.py
kamaki/clients/__init__.py
kamaki/clients/astakos.py
kamaki/clients/compute.py
kamaki/clients/cyclades.py
kamaki/clients/image.py
kamaki/clients/pithos.py...

c270fe96 10/26/2012 04:18 pm Stavros Sachtouris

Remove relative imports from all pkgs, fix astakos

astakos imports an anused (and soon deprecated) module

6ce9fc72 10/25/2012 07:37 pm Stavros Sachtouris

Handle connection.close in repeated requests

a62c75d1 10/23/2012 05:54 pm Stavros Sachtouris

Push error raising upload for testing

19f4195f 10/23/2012 05:21 pm Stavros Sachtouris

Raise block upload greenlet failure error

33e3437a 10/15/2012 12:44 pm Stavros Sachtouris

Quota Holder init (no functionality)

e3f5a1f6 09/20/2012 05:31 pm Stavros Sachtouris

Minor fixes

ebdecdc7 09/20/2012 12:26 pm Stavros Sachtouris

Unitest purge, create_by_manifestation, versionlst

22fc09fb 09/19/2012 05:17 pm Stavros Sachtouris

Support range at file (threaded) download

642f1bbd 09/19/2012 04:43 pm Stavros Sachtouris

Add range in stream download

38dc5d2f 09/19/2012 02:06 pm Stavros Sachtouris

Get rid of pithos_sh

The old profecy "there will be only one [Pithos Client]" is finally
fullfilled

a23f6ffe 09/19/2012 01:42 pm Stavros Sachtouris

Kamaki CLI/clients sharers implementation

mimics pithos_sh account sharers

48c3782c 09/18/2012 06:31 pm Stavros Sachtouris

Get rid of pithos_sh hash methods in download

973fbcad 09/18/2012 06:10 pm Stavros Sachtouris

Much faster download resume

Don't preload local file blocks, but load each one dynamically
only before it is downloaded, and compare

624ee36f 09/18/2012 05:29 pm Stavros Sachtouris

Minor CLI download behavior fixes

20c13fcc 09/18/2012 04:58 pm Stavros Sachtouris

Pithos download supports resume (again)

1785ad41 09/18/2012 03:55 pm Stavros Sachtouris

Voodo-fix mysterious header-destroying request bug

0004301f 09/18/2012 02:07 pm Stavros Sachtouris

Fix safe headers/params bug, pithos download bug

9a7efb0d 09/18/2012 12:11 pm Stavros Sachtouris

Greenlet/thread safe header/param requests

Requests can now distinguish between "global" headers/params
and thread-specific.

fbfee225 09/14/2012 03:50 pm Stavros Sachtouris

Refactor pithos+ download

3-modes:
a sequential
b sequential with resume
c asynchronous/parallel
bug: Still can't multi-download correctly

fb0cd49a 09/13/2012 01:28 pm Stavros Sachtouris

Workarround an eventlist random bug in downloading

699d3bb1 09/13/2012 12:47 pm Stavros Sachtouris

Make object-download code readable

64ab4c13 09/12/2012 06:47 pm Stavros Sachtouris

Mild refactoring to download and upload @pithos.py

f0598cb2 09/12/2012 01:21 pm Stavros Sachtouris

Fix gevent bug, move release action to connection

74275b1a 09/12/2012 12:16 pm Stavros Sachtouris

Minor bugfixes

5b263ba2 09/11/2012 06:10 pm Stavros Sachtouris

Implement kamakicon, use it

kamakicon: an httplib/ObjectPool based connection class

b1713259 09/10/2012 04:30 pm Stavros Sachtouris

Minor bugfixes

6ad245d5 09/07/2012 12:51 pm Stavros Sachtouris

Use connection pool after redesigning http connect

HTTPRequestsXxx objects now use connection pooling
Atually, connections are not pooled, but responses are.

Implementation of connection pooling with httplib not ready yet

Pooling tested in uploads

2f749e6e 08/29/2012 05:26 pm Stavros Sachtouris

Major changes in client, fixed setup

Seperate Client to Client and HTTPConnection
The later is put in a separate package kamaki.client.connection
It manages headers, params, method, url and data of a connection
but the actuall http connection is not implemented at this...

8af4cc0b 08/28/2012 01:27 pm Stavros Sachtouris

Imporove meta methods

e92440bd 08/28/2012 12:17 pm Stavros Sachtouris

Minor bug fixes

d804de82 08/24/2012 05:51 pm Stavros Sachtouris

Add more functionalities to dnl/hashmap/versions

download: progressbar control, overide file, resume (default)
all: extra options for conditional retrieval

d726b3d0 08/24/2012 01:46 pm Stavros Sachtouris

Fixed bug with reset_headers

Now headers are reset after each request by default
but there is an option for not reseting headers, in low
level

65c08a69 08/23/2012 07:35 pm Stavros Sachtouris

Versions in download

56f0908a 08/23/2012 07:15 pm Stavros Sachtouris

Improved incremental download, hashmap, versions

a5e0629d 08/23/2012 11:33 am Stavros Sachtouris

Add full pithos+ move in client.pithos

7d420701 08/22/2012 06:25 pm Stavros Sachtouris

Full pithos+ copy in clients.pithos

2dfcddd8 08/21/2012 04:25 pm Stavros Sachtouris

Correct some typos

65a45524 08/09/2012 06:23 pm Stavros Sachtouris

Add arguments/options to upload

manifest is now a seperated method for uploading
old, naive upload is preserved with --unchunked option

a298f2ab 08/09/2012 03:32 pm Stavros Sachtouris

Add options to store_delete

7234a287 08/09/2012 01:28 pm Stavros Sachtouris

Add options to meta

c3659429 08/08/2012 09:15 pm Stavros Sachtouris

Complete store_list is superset of pithos_sh List

50dff4ea 08/08/2012 07:06 pm Stavros Sachtouris

Add options to store_list

435008b6 08/01/2012 05:48 pm Stavros Sachtouris

Asynchronous uploads with gevent-greenlet

Merge with koukis-fix

0aaad5c7 07/30/2012 06:49 pm Stavros Sachtouris

(Almost) complete pithos client unittesting

transfer_encoding still unchecked

f70616fc 07/27/2012 06:09 pm Stavros Sachtouris

Adjust unit-test object_head/get/put/copy +bugs

Missing:
- account_quota/versioning
- container_post with transfer-encoding
- object_put with transfer-encoding
- object_copy with manifest
- adjust object_move/post/delete

16ce7b91 07/26/2012 03:36 pm Stavros Sachtouris

Adjusting account/container opps + bug fixes

c2236544 07/26/2012 12:04 pm Stavros Sachtouris

Fixed bug with large file uploads

6ff87724 07/25/2012 05:00 pm Stavros Sachtouris

Added test_object_delete

89c2e77b 07/25/2012 04:37 pm Stavros Sachtouris

Added objectcopy/move/post _ minor bugfixes

b2796c01 07/24/2012 07:06 pm Stavros Sachtouris

Started testing object_copy +minor fixes

9f74ca46 07/24/2012 05:37 pm Stavros Sachtouris

Added tests for object_put + minor bug fixes

3e544e5b 07/19/2012 06:27 pm Stavros Sachtouris

More testing in storage

Completed testing at container level, started at object level

379cd4bb 07/18/2012 06:58 pm Stavros Sachtouris

Implemented tests (Pithos+, partial, for now)

Used unittest, implemented tests for all account level Pithos+
operations and some container level operations
TBC

edbee9c8 07/18/2012 03:06 pm Stavros Sachtouris

Pretyfication adjustments

Fixed colors in flavor and image error messages
Pretyfied image client code

4adfa919 07/17/2012 02:59 pm Stavros Sachtouris

Full Pithos+ support at object level

Rest calls HEAD GET PUT COPY MOVE POST DELETE at object level fully
support Pithos+ capabilities by adding generic methods. Adjusted
existing methods to take advantage of these
Not thoroughly tested...

17edd3f4 07/16/2012 04:06 pm Stavros Sachtouris

Full pithos+ support at container level

Full pithos+ support with ne container_<REST> methods
Old methods adjusted to use this functionality
No changes on CLI

4fd88feb 07/16/2012 01:49 pm Stavros Sachtouris

Modified pithos method signatures

Introduced generic account_head/post/get methods that the other
more specific methods call

3a9e54b0 07/16/2012 01:31 pm Stavros Sachtouris

Completed full pithos+ support at account level

Added full support for pithos+ headers and params for account
POST based operations, modified existing methods to take
advantage of this change

b758e547 07/16/2012 12:30 pm Stavros Sachtouris

Added full pithos support to list_containers

In pithos, added full support for optional parameters and headers supported
in Pithos API in list_containers() (account get)

6657ec8c 07/16/2012 12:00 pm Stavros Sachtouris

Pithos level extentions added

get_account_info now supports until parameter, and if-(un)modified-since headers
in request

bcabbc35 07/10/2012 05:09 pm Stavros Sachtouris

Added progress bar

In CLI store_overwrite with prgress bar, added progress bar at store_append
In pithos added generator watcher method support

6de1f262 07/10/2012 05:09 pm Stavros Sachtouris

Fixed and optiomized delmeta for objects

In storage, fixed delmeta which was disfunctional
In pithos added an optimized implementation using the pithos update parameter