My app has to get the xml from the twitter api for which i use libcurl and its functions.. A colleague of mine tells me it would be better to use cURL from within the program itself.. It would make curl a dep. and as I can see from my limited knowledge, probably fork a new child process..
Can anyone here clarify what should be the correct approach with reason?
If its better to use cURL itelf.. why do they even make libcurl?? Can’t everyone just use cURL from within the code itself??
My suggestion is to use wget instead, not libcurl or curl. If you study the curl documentation, curl is a wrapper frontend to libcurl. So for a system process to be able to call curl, libcurl has to be around and well. I was under the impression that we were working on that knowledge. My suggestion involved calling wget from a system call so that you dont have to install libcurl as one of your deps. I can see a few people wondering why there is a 7 Mb dep for a CLI twitter app. Try to keep it simple. One of the points a programmer has to understand is the practicability of the design. As it is, yours is probably the most perfect way of designing the app. My compliments, but with what im suggesting, your app size decreases drastically. You didnt know because you probably already had libcurl installed or got it fast from a DVD or so. But its big for a simple twitter client. Curl is meant for big things, in Mandriva, it is used for segmented downloading of packages (now its been replaced with aria2c) and for its advanced features like NTLM auth inbuilt etc..
hi,
7MB???
Name : libcurl
Arch : x86_64
Version : 7.19.4
Release : 5.fc10
Size : 309 k
Repo : installed
Summary : A library for getting files from web servers
URL : http://curl.haxx.se/
License : MIT
Description: This package provides a way for applications to use FTP, HTTP, Gopher and
: other servers for getting files.
and
Name : libcurl-devel
Arch : x86_64
Version : 7.19.4
Release : 5.fc10
Size : 466 k
Repo : installed
Summary : Files needed for building applications with libcurl
URL : http://curl.haxx.se/
License : MIT
Description: cURL is a tool for getting files from FTP, HTTP, Gopher, Telnet, and
: Dict servers, using any of the supported protocols. The libcurl-devel
: package includes files needed for developing applications which can
: use cURL’s capabilities internally.
libxml2 is only 1.7MB too..
less than an MB here..
well, curl is actually a stand-alone frontend, to provide both inbuilt C functionality and CLI directly. So in principle, you can use both. but consider wget as well…
Depending on the application you make, i’d suggest libsoup, unless you really really know for a afact that you need “advanced” stuff like curl or wget
hi,
Iva already used libcurl and it isnt very complex.. The point here is whether to use the app curl or the library libcurl??
Ankur
Usually stick with anything which provides APIs in the language which you use, rather than execute an external program, then “screen scrape” the output.
IE. libcurl
can you explain this? why stick to the API and not use the command? It would help us understand..
Execing another binary is ok if you need to do it, but why bother? libcurl is there to give your app the functionality you need, execing another app and screen scraping is messy, prone to path issues, and also breaks if the app ever changes its output.
As someone who spends a lot of a time debugging HTTP, I can say tat curl is in another league compared to wget.
urpmi libcurl-devel
To satisfy dependencies, the following packages are going to be installed:
Package Version Release Arch
(medium “Main (Official2008.1-1)”)
libext2fs-devel 1.40.8 1mnb1 i586
libidn-devel 1.5 1mdv2008.1 i586
libsasl2-devel 2.1.22 27mdv2008.1 i586
libssh2-devel 0.18 3mdv2008.1 i586
(medium “Main Updates (Official2008.1-2)”)
curl 7.18.0 1.1mdv2008.1 i586
libcurl-devel 7.18.0 1.1mdv2008.1 i586
libcurl4 7.18.0 1.1mdv2008.1 i586
libkrb53-devel 1.6.3 6.2mdv2008.1 i586
libldap2.4_2 2.4.8 2.1mdv2008.1 i586
libldap2.4_2-devel 2.4.8 2.1mdv2008.1 i586
libopenssl0.9.8 0.9.8g 4.3mdv2008.1 i586
libopenssl0.9.8-devel 0.9.8g 4.3mdv2008.1 i586
openldap 2.4.8 2.1mdv2008.1 i586
openssl 0.9.8g 4.3mdv2008.1 i586
14MB of additional disk space will be used.
6.9MB of packages will be retrieved.
Proceed with the installation of the 14 packages? (Y/n)
as always, the deps spoil the party?
you have the deps installed and thats why its smaller. whats wrong with using wget instead? theres no libwget and obviously wget is more universal…
hi,
Refer the previous posts
I’m saying this again.. You will not need libcurl-devel once an rpm is available.. just libcurl, should have less deps..
http://daniel.haxx.se/docs/curl-vs-wget.html
Ankur
well go ahead use curl then. The “this vs that” page is one of my weaknesses. If you see, everything is a payoff, and i dont like vs pages which begin with that is my “baby”. all i said is:-
i) its possible to use wget
ii) wget is bundled more often with distros than curl
iii) hence curl needs to be downloaded and wget does not have to be….
cURL does not have to be downloaded.. libcurl does.. I’m not using cURL, i’m using the API for C, i.e. libcurl..
While writing an app, the way I think, you should use components that make the app better, more efficient rather than components that will cause “less deps to be downloaded” .. I don’t see the point of trying to reduce the amount of deps.. If a good API is available, use it.
The “vs page” is a factual comparison. I don’t read “this is my baby” anywhere. There are good and bad characteristics of both. You use what you want. I’m already using libcurl and have no intention of changing my choice. The discussion hasn’t really given me *any* reason to..
Ankur
“execing another app and screen scraping is messy, prone to path issues, and also breaks if the app ever changes its output.”
As i see it, wget is going to keep downloading files for the rest of its working future, and it outputs nothing, screen scraping isnt needed. the path is easily accessible due to its being on the sys path
“While writing an app, the way I think, you should use components that make the app better, more efficient rather than components that will cause “less deps to be downloaded” .. I don’t see the point of trying to reduce the amount of deps.. If a good API is available, use it.”
More efficient? What do you base that assertion upon? to make a comment like that and to keep it afloat would mean youve actually timed the processes on similar network conditions and seen the mem usage, speed and other relevant parameters. If you’ve done that, and found libcurl significantly faster, i suggest you put the results up. Nothing like that actually exists at the moment.
libcurl has to be downloaded, wget doesnot. If you can parse a xml file on the disk rather than on the web, you job is done. I think you should be able to do that.
The discussion hasn’t really given me *any* reason to..
you of course are allowed to derive your own conclusions…
“More efficient? What do you base that assertion upon? to make a comment like that and to keep it afloat would mean youve actually timed the processes on similar network conditions and seen the mem usage, speed and other relevant parameters. If you’ve done that, and found libcurl significantly faster, i suggest you put the results up. Nothing like that actually exists at the moment.”
Speed is not the only criteria.. using wget will require a new process.. YES it will..
- libcurl provides error handling , wget does not
- i can channel data from libcurl directly to a function for parsing (im in the process of implementing this, the current “release” uses a temp file) , wget downloads a file which has to be parsed : “you move data from memory to disk and back to memory again”
I just had a chat with folks at ##C who clarified this for me. Using a library is better.. Libcurl it is..
“libcurl has to be downloaded, wget doesnot. If you can parse a xml file on the disk rather than on the web, you job is done. I think you should be able to do that.”
That is not a criteria while writing an app.. “OMG! i cant use this.. it’ll have to be downloaded” ???
“The discussion hasn’t really given me *any* reason to..
you of course are allowed to derive your own conclusions…”
Yes, i certainly am. Seriously, while looking for reasons to use wget instead of libcurl, I ended up finding reasons to stick to libcurl..
Looks like the post has done its job. If more people want to give their views and clarification, please do.
Err, wait, the twitter API isn’t exactly stable. I have a bit of experience with it. So if there is *any* kind of unexpected behavior, you’d be in trouble. For example when the API is under heavy load, sometimes empty json is returned. Also, I don’t think you can check for HTTP status codes returned from the server when you are just system(“wget”)-ing. Expecting the API to behave ideally is just plain wrong.
Right now I found out calling system() does a fork() too. So for now I’ll just pretend to be a know-it-all guy and come to a premature albeit somewhat academically right conclusion that since the library is in memory and returning to the caller process requires some context switching, the former method will be faster. I’m trying to write a quantitative test for this. Let’s see if I will be successful with it.