All entries
Firefox and automatic proxy configuration
Firefox supports automatic proxy configuration, which means that if you plug your laptop on a network with a properly set up proxy server, it will automatically reconfigure itself to use it.
Oddly enough, however, it requires garbage collection when you then plug it back on a network without proxy, in order not to get "proxy not found" errors.
Temporarily disabling file caching
Does it happen to you that you cp a big, big file (say, similar in order of magnitude to the amount of RAM) and the system becomes rather unusable?
It looks like Linux is saying "let's cache this", and as you copy it will try to free more and more ram in order to cache the big file you're copying. In the end, all the RAM is full with file data that you are not going to need.
This varies according to how /proc/sys/vm/swappiness is set.
I learnt about posix_fadvise and I tried to play with it. The result is this
preloadable library that
hooks into open(2) and fadvises everything as POSIX_FADV_DONTNEED.
It is all rather awkward. fadvise in that way will discard existing cache pages if the file is already cached, which is too much. Ideally one would like to say "don't cache this because of me" without stepping on the feet of other system activities.
Also, I found I need to also hook into write(2) and run fadvise after every
single write, because you can't fadvise a file to be written in its entirety,
unless you pass fadvise the file size in advance. But the size of the output
file cannot be known by the preloaded library, so meh.
So, now I can run: nocache cp bigfile someplace/ without trashing the
existing caches. I can also run nocache tar zxf foo.tar.gz and so on.
I wish, of course, that there were no need to do so in the first place.
Here is the nocache library source code, for reference:
/* * nocache - LD_PRELOAD library to fadvise written files to not be cached * * Copyright (C) 2009--2010 Enrico Zini <enrico@enricozini.org> * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #define _XOPEN_SOURCE 600 #include <fcntl.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <dlfcn.h> #include <stdarg.h> #include <errno.h> #include <stdio.h> typedef int (*open_t)(const char*, int, ...); typedef int (*write_t)(int fd, const void *buf, size_t count); int open(const char *pathname, int flags, ...) { static open_t func = 0; int res; if (!func) func = (open_t)dlsym(RTLD_NEXT, "open"); // Note: I wanted to add O_DIRECT, but it imposes restriction on buffer // alignment if (flags & O_CREAT) { va_list ap; va_start(ap, flags); mode_t mode = va_arg(ap, mode_t); res = func(pathname, flags, mode); va_end(ap); } else res = func(pathname, flags); if (res >= 0) { int saved_errno = errno; int z = posix_fadvise(res, 0, 0, POSIX_FADV_DONTNEED); if (z != 0) fprintf(stderr, "Cannot fadvise on %s: %m\n", pathname); errno = saved_errno; } return res; } int write(int fd, const void *buf, size_t count) { static write_t func = 0; int res; if (!func) func = (write_t)dlsym(RTLD_NEXT, "write"); res = func(fd, buf, count); if (res > 0) { int saved_errno = errno; int z = posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED); if (z != 0) fprintf(stderr, "Cannot fadvise during write: %m\n"); errno = saved_errno; } return res; }
Global trends
For some time I have been trying to pinpoint what it is that is brewing in Italy and risks spreading elsewhere, like it happened in the past.
I don't need to be decent to stay in power
While following a train of thought during a political/philosophical lecture I figured that a current growing trend is to have public figures that are more and more indecent.
In Italy it is very hard to find a public figure you can look up to. It is hard to name a politician that is not involved in some shady exchange of favours or some abues of their powers, and we got used to seeing people in power implicated in major corruption scandals, perverted prostitution affairs, or dealings with international criminal organisations.
They do not normally end up in jail, and in fact they keep being very firmly in power, because they manage to stretch or change the laws to get away, or at least to delay trials in order to trigger some statute of limitations.
Is there a pattern here that, although maybe not as clearly defined as in Italy, can be found more or less globally?
Yesterday I thought that this could be such a pattern:
I don't need to be decent to stay in power
If I think of it like that, then it is most definitely not just an Italian phenomenon. If you tell "one doesn't need to be decent to stay in power" to a British, or to a French, I would not expect them to see anything strange with it. We all find it depressing, but we are all used to it.
It is a pattern with repercussions, though: once that becomes normal in a society, it means that people who get to be in power are free to abuse their power much as they want, as long as they are careful enough not to end up in jail. Because, well, nowadays one doesn't need to be decent to stay in power.
I don't need to follow the law to stay in power
That first pattern is already quite well accepted in Italy. So much well accepted, that I think we are starting to see what comes next.
At the end of March we are going to have elections for some regional governors. Funnily enough, in Lazio, the very important region around Rome, the centre-right coalition failed to submit the paperwork on time, and is out of the elections.
It is not just red tape: at some point someone will have to print out the ballots and dispatch them to the voting booths, so one expects to have the coalition logos and the names of the candidates submitted in time, together with signatures supporting the candidates and whatever else the election process needs.
Well, they missed the deadline, they got there after closing time and the building was, well, closed.
It was a fantastic opportunity for a laugh. Memes blossomed on the Italian intarwebs and we now have 2 or 3 new expressions to mean "stupid".
However, now it's hard to tell what is going to happen. On one hand, you can't exclude one of the two major coalitions because of some bureaucratic detail like an office closing time. But on the other hand, several minor coalitions have been excluded in all sorts of past elections because of similar things, and it really would not be fair to start making exceptions now.
But Lombardia, the region around Milan and Emilia Romagna, the one around Bologna, both very, very important, are having similar kinds of problems.
In both regions the previous governors are running again, for the 3rd time in a row, and most likely they legally can't do it, and if elected one can sue and force them to resign, because they have been in power long enough. Lots of paper is being shuffled at the moment to figure if they can get away with it or not.
Oh, and the lawyers of the candidate for the Milan region also managed to get to the tribunal after closing time, but apparently there was still someone inside and they managed to shout loud enough, or somesuch.
Anyway, the situation is getting hot. The Lazio coalition that has been excluded because of their incompetence is now hard at work pushing their potential voters to mount a fracas. Chances are that eventually they'll get away with it, and manage to take part to the election. If that happens, they will likely get close to winning it.
So this seems to be a new pattern that is emerging:
I don't need to follow the law to stay in power
Which, again, is a pattern with quite some repercussions. It is something much more radical than just an issue with morality: it means feudalism, it means we are culturally ready to accept dictatorship.
So, please do me a favour: do not think for a moment that Italy is just a funny place with lemons and tomatoes, and watch out for these patterns emerging around you.
La storia si ripete
Quasi un anno fa:
Spicca naturalmente l’esclusione del rocker Beppe Maniglia con la sua lista “Muscolosa”. A lui sono state contestate “le firme e irregolarita’ nella presentazione della lista”, come spiega il presidente della commissione elettorale, Francesco Russo. In particolare, la lista di candidati per il Consiglio comunale presentata da Maniglia aveva “alcuni nomi cancellati col bianchetto, quindi non c’era certezza sui candidati che effettivamente sono stati presentati ai sottoscrittori”. I nomi cancellati, come spiega la commissione, erano cosi’ tanti che si scendeva sotto il numero minimo di 31.
Escluse, invece, [...], e Maniglia Muscolosa, di Beppe Maniglia, il musicista che da anni anima i pomeriggi dei bolognesi in piazza Maggiore. La sua lista è stata scartata anche per irregolarità su quattro nomi di candidati: cancellati con un bianchetto e sostituiti a penna con altri quattro, di fatto hanno invalidato complessivamente la lista che, essendo formata da 31 candidati (numero minimo) si è così ridotta a 27 nomi.
Oggi:
http://www.repubblica.it/politica/2010/03/01/news/liste_pdl_nel_lazio-2466722/
Minacciato o solo affamato? O forse impegnato a modifiche di lista dell'ultimo minuto? La vicenda della mancata presentazione delle liste Pdl per le regionali del Lazio nella provincia di Roma oscilla tra la farsa e il dramma politico. Al centro c'è Alfredo Milioni, presidente Pdl del XIX municipio della capitale: l'uomo che ha lasciato l'edificio del Tribunale forse per andare a mangiare un panino o, forse, per cancellare qualche nome dalle liste che stava per presentare su indicazioni, pare, venute molto dall'alto.
Dica la verità.
«Lo giuro, lo giuro! Non volevo apportare modifiche. Mi deve credere, capito?».
Va bene, stia calmo. Questo però significa che è davvero andato a mangiarsi un panino.
«Sì... ecco, sì: sono andato a mangiarmi una panino. Non mi pare grave, no?».
Quindi è vero: lei ha lasciato l’aula per andare al bar.
«Io? A mangiare?».
In conferenza stampa, la Polverini ha fornito una ricostruzione dei fatti un poco diversa.
«No, cioè... io, a mangiare: ma chi l’ha detto?».
Lei, adesso.
Sebbene il buon Beppe Maniglia non meriti davvero di essere messo di fianco a gente oscena di questo tipo.
Cropping images with GDAL
I am working to get a better integration between Meteosatlib and GDAL.
A nice aspect of GDAL is that it allows to create read/write drivers around two
functions: Open and CreateCopy.
Open opens a datset read only, and to implement that all you have to do is to
implement read access to your data using the
GDALDataset interface.
To implement CreateCopy, all you have to do is to create a new file reading
information from a GDALDataset; then call Open on it. This means that there
is no need to support incremental updates, and that all the data required to
create a new file is readily available. This simplifies matters a lot.
GDAL provides some interesting image manipulation functions, that can work over just these two calls. The way it does it is by exploiting the concept of a virtual dataset, which wraps an existing dataset but changes some parameters on the fly.
This means that you can wrap a read only dataset with a virtual dataset that
transforms it somehow, and then pass the virtual dataset to CreateCopy to
save the transformed image in a format of your choice.
On top of that, for many transformations you do not need to create your own virtual datasets, but you can use the functions provided by the VRT GDAL driver.
One can learn a lot on how to use VRTDataset by reading the source code for
gdal_translate. By doing that I could come up with this code for cropping an
image, which is an interesting VRTDataset example.
GDALDataset* crop(GDALDataset* poDS, int xoff, int yoff, int xsize, int ysize) { VRTDataset *poVDS = (VRTDataset*)VRTCreate(xsize, ysize); // Copy dataset info const char* pszProjection = poDS->GetProjectionRef(); if (pszProjection != NULL && strlen(pszProjection) > 0) poVDS->SetProjection(pszProjection); double adfGeoTransform[6]; if (poDS->GetGeoTransform(adfGeoTransform) == CE_None) { // Adapt the geotransform matrix to the subarea adfGeoTransform[0] += xoff * adfGeoTransform[1] + yoff * adfGeoTransform[2]; adfGeoTransform[3] += xoff * adfGeoTransform[4] + yoff * adfGeoTransform[5]; poVDS->SetGeoTransform(adfGeoTransform); } poVDS->SetMetadata(poDS->GetMetadata()); // Here I also copy metadata from my own domain char **papszMD; papszMD = poDS->GetMetadata(MD_DOMAIN_MSAT); if (papszMD != NULL) poVDS->SetMetadata(papszMD, MD_DOMAIN_MSAT); for (int i = 0; i < poDS->GetRasterCount(); ++i) { GDALRasterBand* poSrcBand = poDS->GetRasterBand(i + 1); GDALDataType eBandType = poSrcBand->GetRasterDataType(); poVDS->AddBand(eBandType, NULL); VRTSourcedRasterBand* poVRTBand = (VRTSourcedRasterBand*)poVDS->GetRasterBand(i + 1); poVRTBand->AddSimpleSource(poSrcBand, xoff, yoff, xsize, ysize, 0, 0, xsize, ysize); poVRTBand->CopyCommonInfoFrom(poSrcBand); // Again, I copy my own metadata papszMD = poSrcBand->GetMetadata(MD_DOMAIN_MSAT); if (papszMD != NULL) poVRTBand->SetMetadata(papszMD, MD_DOMAIN_MSAT); } return poVDS; }
This function wraps a dataset with a virtual dataset that crops it. Just pass
the resulting dataset to GDALCreateCopy to save it in the format that you
need.
Custom function decorators with TurboGears 2
I am exposing some library functions using a TurboGears2 controller (see web-api-with-turbogears2). It turns out that some functions return a dict, some a list, some a string, and TurboGears 2 only allows JSON serialisation for dicts.
A simple work-around for this is to wrap the function result into a dict, something like this:
@expose("json") @validate(validator_dispatcher, error_handler=api_validation_error) def list_colours(self, filter=None, productID=None, maxResults=100, **kw): # Call API res = self.engine.list_colours(filter, productID, maxResults) # Return result return dict(r=res)
It would be nice, however, to have an @webapi() decorator that
automatically wraps the function result with the dict:
def webapi(func): def dict_wrap(*args, **kw): return dict(r=func(*args, **kw)) return dict_wrap # ...in the controller... @expose("json") @validate(validator_dispatcher, error_handler=api_validation_error) @webapi def list_colours(self, filter=None, productID=None, maxResults=100, **kw): # Call API res = self.engine.list_colours(filter, productID, maxResults) # Return result return res
This works, as long as @webapi appears last in the list of decorators.
This is because if it appears last it will be the first to wrap the function,
and so it will not interfere with the tg.decorators machinery.
Would it be possible to create a decorator that can be put anywhere among the decorator list? Yes, it is possible but tricky, and it gives me the feeling that it may break in any future version of TurboGears:
class webapi(object): def __call__(self, func): def dict_wrap(*args, **kw): return dict(r=func(*args, **kw)) # Migrate the decoration attribute to our new function if hasattr(func, 'decoration'): dict_wrap.decoration = func.decoration dict_wrap.decoration.controller = dict_wrap delattr(func, 'decoration') return dict_wrap # ...in the controller... @expose("json") @validate(validator_dispatcher, error_handler=api_validation_error) @webapi def list_colours(self, filter=None, productID=None, maxResults=100, **kw): # Call API res = self.engine.list_colours(filter, productID, maxResults) # Return result return res
As a convenience, TurboGears 2 offers, in the decorators module, a way to
build decorator "hooks":
class before_validate(_hook_decorator): '''A list of callables to be run before validation is performed''' hook_name = 'before_validate' class before_call(_hook_decorator): '''A list of callables to be run before the controller method is called''' hook_name = 'before_call' class before_render(_hook_decorator): '''A list of callables to be run before the template is rendered''' hook_name = 'before_render' class after_render(_hook_decorator): '''A list of callables to be run after the template is rendered. Will be run before it is returned returned up the WSGI stack''' hook_name = 'after_render'
The way these are invoked can be found in the _perform_call function in
tg/controllers.py.
To show an example use of those hooks, let's add a some polygen wisdom to every data structure we return:
class wisdom(decorators.before_render): def __init__(self, grammar): super(wisdom, self).__init__(self.add_wisdom) self.grammar = grammar def add_wisdom(self, remainder, params, output): from subprocess import Popen, PIPE output["wisdom"] = Popen(["polyrun", self.grammar], stdout=PIPE).communicate()[0] # ...in the controller... @wisdom("genius") @expose("json") @validate(validator_dispatcher, error_handler=api_validation_error) def list_colours(self, filter=None, productID=None, maxResults=100, **kw): # Call API res = self.engine.list_colours(filter, productID, maxResults) # Return result return res
These hooks cannot however be used for what I need, that is, to wrap the result inside a dict. The reason is because they are called in this way:
controller.decoration.run_hooks( 'before_render', remainder, params, output)
and not in this way:
output = controller.decoration.run_hooks( 'before_render', remainder, params, output)
So it is possible to modify the output (if it is a mutable structure) but not to exchange it with something else.
Can we do even better? Sure we can. We can assimilate @expose and @validate
inside @webapi to avoid repeating those same many decorator lines over and
over again:
class webapi(object): def __init__(self, error_handler = None): self.error_handler = error_handler def __call__(self, func): def dict_wrap(*args, **kw): return dict(r=func(*args, **kw)) res = expose("json")(dict_wrap) res = validate(validator_dispatcher, error_handler=self.error_handler)(res) return res # ...in the controller... @expose("json") def api_validation_error(self, **kw): pylons.response.status = "400 Error" return dict(e="validation error on input fields", form_errors=pylons.c.form_errors) @webapi(error_handler=api_validation_error) def list_colours(self, filter=None, productID=None, maxResults=100, **kw): # Call API res = self.engine.list_colours(filter, productID, maxResults) # Return result return res
This got rid of @expose and @validate, and provides almost all the
default values that I need. Unfortunately I could not find out how to access
api_validation_error from the decorator so that I can pass it to the
validator, therefore I remain with the inconvenience of having to explicitly
pass it every time.
Released cfget 0.8
I have released version 0.8 of cfget.
I worked on 3 things:
- A big cleanup with the excuse of uploading to Debian
--root=pathto limit queries and dumps to all keys under a given path- when used in template mode with an output file name, it will write it atomically.