Discussion:
Improved dependency management in RCP/Tycho - (request for comments)
(too old to reply)
Tom Bujok
2012-08-20 15:38:02 UTC
Permalink
Hi guys,

I am an experienced java developer who was "thrown" into the backend layer of an Eclipse RCP project some time ago. At first, I thought I was gonna enjoy it, since my expertise includes OSGi, but I was shocked to end up in an "un-mavenized" piece of work with one eclipse-plugin per external dependency. If you wanted to add an additional dependency an extra eclipse plugin had to be created, jar had to be stored in the lib folder (in the SVN as well), and what is more, MANIFEST.MF had to be created manually by hand! (sic!) At some companies that I know, that are even special job positions (called dependency configurators) where there is a special guy who's responsibility is to add a dependency to a RCP project whenever a new ticket is created…

Cutting the long story short, I was shocked how difficult it was to fully "mavenize" the project and to configure a proper dependency management in Eclipse RCP together with Tycho.
The problems that I met:
1.) not all dependencies are located in well-known p2 update sites
2.) not all dependencies are OSGi bundles
3.) pomDependencies consider -> solves the problem half-way only, since dependencies defined in such a way cannot be referenced in the target platform so the project does not compile in the Eclipse IDE
4.) if I use pomDependencies->consider I cannot include two versions of the same dependency (standard maven mechanism)
5.) then I (of course) played with the maven-bundle-plugin. I created a separate 3rd party project to automate it a bit, but it creates one bundle for all dependencies, it's not that flexible and its build cannot be included in the same reactor build than the Eclipse-RCP project itself, so the builds had to be invoked one after the other. I somehow managed to import it to Eclipse IDE so that everything works, but I got 10 calls a day from people who used to get compilation problems after adding a singe dependency to that construct (sic^2)

My conclusion was:
- pom first projects sucks (problems described above)
- manifest-first projects would rock, if only we had all dependencies in a generally available p2 site (which we of course did not have)

Basically, I read a lot of forums, blogs, etc, and people keep struggling with relatively simple things (that I have described above) all the time.
Here's a blog entry listing the problems (posted by somebody else): http://bit.ly/PypQEy
I followed the same path almost one year ago.

So, I decided to contribute a bit to the great job that you are doing with Tycho and I create a p2-maven-plugin - he link is here -> https://github.com/reficio/p2-maven-plugin
It was supposed to be a remedy to the Eclipse-RCP/Maven/Tycho/IDE dependency management problems.
It's main goal is to:
- download all required dependencies (from maven repositories) to a local folder
- recognize which dependencies are not OSGi bundles and bundle them using the "bnd" tool (the ones that are bundles are simply copied)
- take all the files (now bundles) and invoke a P2 tool to generate the P2 site (with categories) that can be directly consumed by the Target Definition in the Eclipse IDE
I think that I create something that was described by Igor Fedorenko in this post: http://www.eclipse.org/forums/index.php/mv/msg/6192/18769/#msg_18769

Please see some example, how to use it: the following snippet will generate a sample p2-site (fetching all transitive dependencies and bundling jars that are not OSGi bundles).
<plugin>
<groupId>org.reficio</groupId>
<artifactId>p2-maven-plugin</artifactId>
<version>1.0.0-SNAPSHOT</version>
<executions>
<execution>
<id>default-cli</id>
<configuration>
<artifacts>
<artifact><id>commons-io:commons-io:2.1</id></artifact>
<artifact><id>commons-lang:commons-lang:2.4</id></artifact>
<artifact><id>commons-lang:commons-lang:2.5</id></artifact>
<artifact><id>commons-lang:commons-lang:2.6</id></artifact>
<artifact><id>org.apache.commons:commons-lang3:3.1</id></artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>

The fetch of transitive dependencies may be disabled using the <transitive>false</transitive> option on the <artifcat> level.
The MANIFEST.MF directives may be specified using the <instructions>, known from the maven-bundle-plugin:
<artifact>
<id>commons-io:commons-io:2.1</id>
<transitive>false</transitive>
<instructions>
<Import-Package>*;resolution:=optional</Import-Package>
<Export-Package>*</Export-Package>
(…)
</instructions>
</artifact>

Finally, the default MANIFEST.MF can be regenerated and overwritten (specific Imports/Exports may be then specified) using the <override>true</override> option .
Then the generated site may be exposed using a jetty-plugin and consumed in the Eclipse IDE.
Please, see the examples section (I described everything there): https://github.com/reficio/p2-maven-plugin#examples

I would like to ask you a few questions, since you guys are developers of Tycho:
- How do you like that plugin? Do you think it is of any good?
- What should be improved?

I am willing to contribute - and I think that the whole mechanism may be improved further. I was trying to invoke the plugin automatically before Tycho starts its project configurators:
http://dev.eclipse.org/mhonarc/lists/tycho-dev/msg00541.html
http://dev.eclipse.org/mhonarc/lists/tycho-user/msg02103.html
but because of some Maven/Plexus limitations it was not that simple. It's impossible now to invoke any Maven logic before Tycho, there are some problems with sharing classes across class-loaders.

My perfect case would be if the plugin was invoked seamlessly by Tycho during the compilation, so that the generated p2-site is automatically added to the Tycho repositories.
Then the perfect case would be if this repository could be referenced from Eclipse IDE, so that the project compiles in the IDE without any duplication in the dependency handling.
Right now, it's not that simple since the local-folder "repository" is not handled properly.

I would be very grateful for any comments.

Regards,

Tom Bujok
http://www.reficio.org
Mikaël Barbero
2012-08-20 15:48:34 UTC
Permalink
Hi Tom,

FYI, Eclipse projects handle your issue of needing external jars through Orbit http://www.eclipse.org/orbit/ . Any jar cannot be included in orbit because it has to pass IP checks. There is a p2 update site that you can add to your target platform on the download page http://download.eclipse.org/tools/orbit/downloads/ . I understand you are willing to solve the issue more globally but it may helps in short terms.

Best regards,
Mikael
Post by Tom Bujok
Hi guys,
I am an experienced java developer who was "thrown" into the backend layer of an Eclipse RCP project some time ago. At first, I thought I was gonna enjoy it, since my expertise includes OSGi, but I was shocked to end up in an "un-mavenized" piece of work with one eclipse-plugin per external dependency. If you wanted to add an additional dependency an extra eclipse plugin had to be created, jar had to be stored in the lib folder (in the SVN as well), and what is more, MANIFEST.MF had to be created manually by hand! (sic!) At some companies that I know, that are even special job positions (called dependency configurators) where there is a special guy who's responsibility is to add a dependency to a RCP project whenever a new ticket is created…
Cutting the long story short, I was shocked how difficult it was to fully "mavenize" the project and to configure a proper dependency management in Eclipse RCP together with Tycho.
1.) not all dependencies are located in well-known p2 update sites
2.) not all dependencies are OSGi bundles
3.) pomDependencies consider -> solves the problem half-way only, since dependencies defined in such a way cannot be referenced in the target platform so the project does not compile in the Eclipse IDE
4.) if I use pomDependencies->consider I cannot include two versions of the same dependency (standard maven mechanism)
5.) then I (of course) played with the maven-bundle-plugin. I created a separate 3rd party project to automate it a bit, but it creates one bundle for all dependencies, it's not that flexible and its build cannot be included in the same reactor build than the Eclipse-RCP project itself, so the builds had to be invoked one after the other. I somehow managed to import it to Eclipse IDE so that everything works, but I got 10 calls a day from people who used to get compilation problems after adding a singe dependency to that construct (sic^2)
- pom first projects sucks (problems described above)
- manifest-first projects would rock, if only we had all dependencies in a generally available p2 site (which we of course did not have)
Basically, I read a lot of forums, blogs, etc, and people keep struggling with relatively simple things (that I have described above) all the time.
Here's a blog entry listing the problems (posted by somebody else): http://bit.ly/PypQEy
I followed the same path almost one year ago.
So, I decided to contribute a bit to the great job that you are doing with Tycho and I create a p2-maven-plugin - he link is here -> https://github.com/reficio/p2-maven-plugin
It was supposed to be a remedy to the Eclipse-RCP/Maven/Tycho/IDE dependency management problems.
- download all required dependencies (from maven repositories) to a local folder
- recognize which dependencies are not OSGi bundles and bundle them using the "bnd" tool (the ones that are bundles are simply copied)
- take all the files (now bundles) and invoke a P2 tool to generate the P2 site (with categories) that can be directly consumed by the Target Definition in the Eclipse IDE
I think that I create something that was described by Igor Fedorenko in this post: http://www.eclipse.org/forums/index.php/mv/msg/6192/18769/#msg_18769
Please see some example, how to use it: the following snippet will generate a sample p2-site (fetching all transitive dependencies and bundling jars that are not OSGi bundles).
<plugin>
<groupId>org.reficio</groupId>
<artifactId>p2-maven-plugin</artifactId>
<version>1.0.0-SNAPSHOT</version>
<executions>
<execution>
<id>default-cli</id>
<configuration>
<artifacts>
<artifact><id>commons-io:commons-io:2.1</id></artifact>
<artifact><id>commons-lang:commons-lang:2.4</id></artifact>
<artifact><id>commons-lang:commons-lang:2.5</id></artifact>
<artifact><id>commons-lang:commons-lang:2.6</id></artifact>
<artifact><id>org.apache.commons:commons-lang3:3.1</id></artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
The fetch of transitive dependencies may be disabled using the <transitive>false</transitive> option on the <artifcat> level.
<artifact>
<id>commons-io:commons-io:2.1</id>
<transitive>false</transitive>
<instructions>
<Import-Package>*;resolution:=optional</Import-Package>
<Export-Package>*</Export-Package>
(…)
</instructions>
</artifact>
Finally, the default MANIFEST.MF can be regenerated and overwritten (specific Imports/Exports may be then specified) using the <override>true</override> option .
Then the generated site may be exposed using a jetty-plugin and consumed in the Eclipse IDE.
Please, see the examples section (I described everything there): https://github.com/reficio/p2-maven-plugin#examples
- How do you like that plugin? Do you think it is of any good?
- What should be improved?
http://dev.eclipse.org/mhonarc/lists/tycho-dev/msg00541.html
http://dev.eclipse.org/mhonarc/lists/tycho-user/msg02103.html
but because of some Maven/Plexus limitations it was not that simple. It's impossible now to invoke any Maven logic before Tycho, there are some problems with sharing classes across class-loaders.
My perfect case would be if the plugin was invoked seamlessly by Tycho during the compilation, so that the generated p2-site is automatically added to the Tycho repositories.
Then the perfect case would be if this repository could be referenced from Eclipse IDE, so that the project compiles in the IDE without any duplication in the dependency handling.
Right now, it's not that simple since the local-folder "repository" is not handled properly.
I would be very grateful for any comments.
Regards,
Tom Bujok
http://www.reficio.org
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
Tom Bujok
2012-08-20 15:56:37 UTC
Permalink
Hi Mikael,

Thanks for your quick response. But, as you said, I am trying to solve it more globally. There are internal corporate jars that cannot be uploaded anywhere.
My plugins automates a lot, but there's one missing step in the integration between my plugin and Tycho to get rid of jetty and make the integration seamless.

Tom
Post by Mikaël Barbero
Hi Tom,
FYI, Eclipse projects handle your issue of needing external jars through Orbit http://www.eclipse.org/orbit/ . Any jar cannot be included in orbit because it has to pass IP checks. There is a p2 update site that you can add to your target platform on the download page http://download.eclipse.org/tools/orbit/downloads/ . I understand you are willing to solve the issue more globally but it may helps in short terms.
Best regards,
Mikael
Post by Tom Bujok
Hi guys,
I am an experienced java developer who was "thrown" into the backend layer of an Eclipse RCP project some time ago. At first, I thought I was gonna enjoy it, since my expertise includes OSGi, but I was shocked to end up in an "un-mavenized" piece of work with one eclipse-plugin per external dependency. If you wanted to add an additional dependency an extra eclipse plugin had to be created, jar had to be stored in the lib folder (in the SVN as well), and what is more, MANIFEST.MF had to be created manually by hand! (sic!) At some companies that I know, that are even special job positions (called dependency configurators) where there is a special guy who's responsibility is to add a dependency to a RCP project whenever a new ticket is created…
Cutting the long story short, I was shocked how difficult it was to fully "mavenize" the project and to configure a proper dependency management in Eclipse RCP together with Tycho.
1.) not all dependencies are located in well-known p2 update sites
2.) not all dependencies are OSGi bundles
3.) pomDependencies consider -> solves the problem half-way only, since dependencies defined in such a way cannot be referenced in the target platform so the project does not compile in the Eclipse IDE
4.) if I use pomDependencies->consider I cannot include two versions of the same dependency (standard maven mechanism)
5.) then I (of course) played with the maven-bundle-plugin. I created a separate 3rd party project to automate it a bit, but it creates one bundle for all dependencies, it's not that flexible and its build cannot be included in the same reactor build than the Eclipse-RCP project itself, so the builds had to be invoked one after the other. I somehow managed to import it to Eclipse IDE so that everything works, but I got 10 calls a day from people who used to get compilation problems after adding a singe dependency to that construct (sic^2)
- pom first projects sucks (problems described above)
- manifest-first projects would rock, if only we had all dependencies in a generally available p2 site (which we of course did not have)
Basically, I read a lot of forums, blogs, etc, and people keep struggling with relatively simple things (that I have described above) all the time.
Here's a blog entry listing the problems (posted by somebody else): http://bit.ly/PypQEy
I followed the same path almost one year ago.
So, I decided to contribute a bit to the great job that you are doing with Tycho and I create a p2-maven-plugin - he link is here -> https://github.com/reficio/p2-maven-plugin
It was supposed to be a remedy to the Eclipse-RCP/Maven/Tycho/IDE dependency management problems.
- download all required dependencies (from maven repositories) to a local folder
- recognize which dependencies are not OSGi bundles and bundle them using the "bnd" tool (the ones that are bundles are simply copied)
- take all the files (now bundles) and invoke a P2 tool to generate the P2 site (with categories) that can be directly consumed by the Target Definition in the Eclipse IDE
I think that I create something that was described by Igor Fedorenko in this post: http://www.eclipse.org/forums/index.php/mv/msg/6192/18769/#msg_18769
Please see some example, how to use it: the following snippet will generate a sample p2-site (fetching all transitive dependencies and bundling jars that are not OSGi bundles).
<plugin>
<groupId>org.reficio</groupId>
<artifactId>p2-maven-plugin</artifactId>
<version>1.0.0-SNAPSHOT</version>
<executions>
<execution>
<id>default-cli</id>
<configuration>
<artifacts>
<artifact><id>commons-io:commons-io:2.1</id></artifact>
<artifact><id>commons-lang:commons-lang:2.4</id></artifact>
<artifact><id>commons-lang:commons-lang:2.5</id></artifact>
<artifact><id>commons-lang:commons-lang:2.6</id></artifact>
<artifact><id>org.apache.commons:commons-lang3:3.1</id></artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
The fetch of transitive dependencies may be disabled using the <transitive>false</transitive> option on the <artifcat> level.
<artifact>
<id>commons-io:commons-io:2.1</id>
<transitive>false</transitive>
<instructions>
<Import-Package>*;resolution:=optional</Import-Package>
<Export-Package>*</Export-Package>
(…)
</instructions>
</artifact>
Finally, the default MANIFEST.MF can be regenerated and overwritten (specific Imports/Exports may be then specified) using the <override>true</override> option .
Then the generated site may be exposed using a jetty-plugin and consumed in the Eclipse IDE.
Please, see the examples section (I described everything there): https://github.com/reficio/p2-maven-plugin#examples
- How do you like that plugin? Do you think it is of any good?
- What should be improved?
http://dev.eclipse.org/mhonarc/lists/tycho-dev/msg00541.html
http://dev.eclipse.org/mhonarc/lists/tycho-user/msg02103.html
but because of some Maven/Plexus limitations it was not that simple. It's impossible now to invoke any Maven logic before Tycho, there are some problems with sharing classes across class-loaders.
My perfect case would be if the plugin was invoked seamlessly by Tycho during the compilation, so that the generated p2-site is automatically added to the Tycho repositories.
Then the perfect case would be if this repository could be referenced from Eclipse IDE, so that the project compiles in the IDE without any duplication in the dependency handling.
Right now, it's not that simple since the local-folder "repository" is not handled properly.
I would be very grateful for any comments.
Regards,
Tom Bujok
http://www.reficio.org
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
Jason van Zyl
2012-08-20 16:35:59 UTC
Permalink
The best way to handle dependencies from a Maven repository is with what we call wrapper bundles. It's an integration between Maven and Tycho that leverages the BND and the Maven Bundle Plugin. If you're interested you can take a look at m2eclipse itself. We spent quite a bit of time patching Maven, Tycho, BND and the Maven bundle plugins to make creating bundles from dependencies from Maven repositories.
Post by Tom Bujok
Hi guys,
I am an experienced java developer who was "thrown" into the backend layer of an Eclipse RCP project some time ago. At first, I thought I was gonna enjoy it, since my expertise includes OSGi, but I was shocked to end up in an "un-mavenized" piece of work with one eclipse-plugin per external dependency. If you wanted to add an additional dependency an extra eclipse plugin had to be created, jar had to be stored in the lib folder (in the SVN as well), and what is more, MANIFEST.MF had to be created manually by hand! (sic!) At some companies that I know, that are even special job positions (called dependency configurators) where there is a special guy who's responsibility is to add a dependency to a RCP project whenever a new ticket is created…
Cutting the long story short, I was shocked how difficult it was to fully "mavenize" the project and to configure a proper dependency management in Eclipse RCP together with Tycho.
1.) not all dependencies are located in well-known p2 update sites
2.) not all dependencies are OSGi bundles
3.) pomDependencies consider -> solves the problem half-way only, since dependencies defined in such a way cannot be referenced in the target platform so the project does not compile in the Eclipse IDE
4.) if I use pomDependencies->consider I cannot include two versions of the same dependency (standard maven mechanism)
5.) then I (of course) played with the maven-bundle-plugin. I created a separate 3rd party project to automate it a bit, but it creates one bundle for all dependencies, it's not that flexible and its build cannot be included in the same reactor build than the Eclipse-RCP project itself, so the builds had to be invoked one after the other. I somehow managed to import it to Eclipse IDE so that everything works, but I got 10 calls a day from people who used to get compilation problems after adding a singe dependency to that construct (sic^2)
- pom first projects sucks (problems described above)
- manifest-first projects would rock, if only we had all dependencies in a generally available p2 site (which we of course did not have)
Basically, I read a lot of forums, blogs, etc, and people keep struggling with relatively simple things (that I have described above) all the time.
Here's a blog entry listing the problems (posted by somebody else): http://bit.ly/PypQEy
I followed the same path almost one year ago.
So, I decided to contribute a bit to the great job that you are doing with Tycho and I create a p2-maven-plugin - he link is here -> https://github.com/reficio/p2-maven-plugin
It was supposed to be a remedy to the Eclipse-RCP/Maven/Tycho/IDE dependency management problems.
- download all required dependencies (from maven repositories) to a local folder
- recognize which dependencies are not OSGi bundles and bundle them using the "bnd" tool (the ones that are bundles are simply copied)
- take all the files (now bundles) and invoke a P2 tool to generate the P2 site (with categories) that can be directly consumed by the Target Definition in the Eclipse IDE
I think that I create something that was described by Igor Fedorenko in this post: http://www.eclipse.org/forums/index.php/mv/msg/6192/18769/#msg_18769
Please see some example, how to use it: the following snippet will generate a sample p2-site (fetching all transitive dependencies and bundling jars that are not OSGi bundles).
<plugin>
<groupId>org.reficio</groupId>
<artifactId>p2-maven-plugin</artifactId>
<version>1.0.0-SNAPSHOT</version>
<executions>
<execution>
<id>default-cli</id>
<configuration>
<artifacts>
<artifact><id>commons-io:commons-io:2.1</id></artifact>
<artifact><id>commons-lang:commons-lang:2.4</id></artifact>
<artifact><id>commons-lang:commons-lang:2.5</id></artifact>
<artifact><id>commons-lang:commons-lang:2.6</id></artifact>
<artifact><id>org.apache.commons:commons-lang3:3.1</id></artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
The fetch of transitive dependencies may be disabled using the <transitive>false</transitive> option on the <artifcat> level.
<artifact>
<id>commons-io:commons-io:2.1</id>
<transitive>false</transitive>
<instructions>
<Import-Package>*;resolution:=optional</Import-Package>
<Export-Package>*</Export-Package>
(…)
</instructions>
</artifact>
Finally, the default MANIFEST.MF can be regenerated and overwritten (specific Imports/Exports may be then specified) using the <override>true</override> option .
Then the generated site may be exposed using a jetty-plugin and consumed in the Eclipse IDE.
Please, see the examples section (I described everything there): https://github.com/reficio/p2-maven-plugin#examples
- How do you like that plugin? Do you think it is of any good?
- What should be improved?
http://dev.eclipse.org/mhonarc/lists/tycho-dev/msg00541.html
http://dev.eclipse.org/mhonarc/lists/tycho-user/msg02103.html
but because of some Maven/Plexus limitations it was not that simple. It's impossible now to invoke any Maven logic before Tycho, there are some problems with sharing classes across class-loaders.
My perfect case would be if the plugin was invoked seamlessly by Tycho during the compilation, so that the generated p2-site is automatically added to the Tycho repositories.
Then the perfect case would be if this repository could be referenced from Eclipse IDE, so that the project compiles in the IDE without any duplication in the dependency handling.
Right now, it's not that simple since the local-folder "repository" is not handled properly.
I would be very grateful for any comments.
Regards,
Tom Bujok
http://www.reficio.org
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
Thanks,

Jason

----------------------------------------------------------
Jason van Zyl
Founder & CTO, Sonatype
Founder, Apache Maven
http://twitter.com/jvanzyl
---------------------------------------------------------

Simplex sigillum veri. (Simplicity is the seal of truth.)







Thanks,

Jason

----------------------------------------------------------
Jason van Zyl
Founder & CTO, Sonatype
Founder, Apache Maven
http://twitter.com/jvanzyl
---------------------------------------------------------

Three people can keep a secret provided two of them are dead.

-- Benjamin Franklin
Tom Bujok
2012-08-20 19:52:27 UTC
Permalink
Hi,

Thanks for your reply. AFAIK it will not work in this use-case:
- create a tycho RCP project
- declare a Maven dependency in pom.xml (with pomDependencies set to consider)
- import project to Eclipse IDE (with m2Eclipse)

Or am I wrong?

Tom
Post by Jason van Zyl
The best way to handle dependencies from a Maven repository is with what we call wrapper bundles. It's an integration between Maven and Tycho that leverages the BND and the Maven Bundle Plugin. If you're interested you can take a look at m2eclipse itself. We spent quite a bit of time patching Maven, Tycho, BND and the Maven bundle plugins to make creating bundles from dependencies from Maven repositories.
Post by Tom Bujok
Hi guys,
I am an experienced java developer who was "thrown" into the backend layer of an Eclipse RCP project some time ago. At first, I thought I was gonna enjoy it, since my expertise includes OSGi, but I was shocked to end up in an "un-mavenized" piece of work with one eclipse-plugin per external dependency. If you wanted to add an additional dependency an extra eclipse plugin had to be created, jar had to be stored in the lib folder (in the SVN as well), and what is more, MANIFEST.MF had to be created manually by hand! (sic!) At some companies that I know, that are even special job positions (called dependency configurators) where there is a special guy who's responsibility is to add a dependency to a RCP project whenever a new ticket is created…
Cutting the long story short, I was shocked how difficult it was to fully "mavenize" the project and to configure a proper dependency management in Eclipse RCP together with Tycho.
1.) not all dependencies are located in well-known p2 update sites
2.) not all dependencies are OSGi bundles
3.) pomDependencies consider -> solves the problem half-way only, since dependencies defined in such a way cannot be referenced in the target platform so the project does not compile in the Eclipse IDE
4.) if I use pomDependencies->consider I cannot include two versions of the same dependency (standard maven mechanism)
5.) then I (of course) played with the maven-bundle-plugin. I created a separate 3rd party project to automate it a bit, but it creates one bundle for all dependencies, it's not that flexible and its build cannot be included in the same reactor build than the Eclipse-RCP project itself, so the builds had to be invoked one after the other. I somehow managed to import it to Eclipse IDE so that everything works, but I got 10 calls a day from people who used to get compilation problems after adding a singe dependency to that construct (sic^2)
- pom first projects sucks (problems described above)
- manifest-first projects would rock, if only we had all dependencies in a generally available p2 site (which we of course did not have)
Basically, I read a lot of forums, blogs, etc, and people keep struggling with relatively simple things (that I have described above) all the time.
Here's a blog entry listing the problems (posted by somebody else): http://bit.ly/PypQEy
I followed the same path almost one year ago.
So, I decided to contribute a bit to the great job that you are doing with Tycho and I create a p2-maven-plugin - he link is here -> https://github.com/reficio/p2-maven-plugin
It was supposed to be a remedy to the Eclipse-RCP/Maven/Tycho/IDE dependency management problems.
- download all required dependencies (from maven repositories) to a local folder
- recognize which dependencies are not OSGi bundles and bundle them using the "bnd" tool (the ones that are bundles are simply copied)
- take all the files (now bundles) and invoke a P2 tool to generate the P2 site (with categories) that can be directly consumed by the Target Definition in the Eclipse IDE
I think that I create something that was described by Igor Fedorenko in this post: http://www.eclipse.org/forums/index.php/mv/msg/6192/18769/#msg_18769
Please see some example, how to use it: the following snippet will generate a sample p2-site (fetching all transitive dependencies and bundling jars that are not OSGi bundles).
<plugin>
<groupId>org.reficio</groupId>
<artifactId>p2-maven-plugin</artifactId>
<version>1.0.0-SNAPSHOT</version>
<executions>
<execution>
<id>default-cli</id>
<configuration>
<artifacts>
<artifact><id>commons-io:commons-io:2.1</id></artifact>
<artifact><id>commons-lang:commons-lang:2.4</id></artifact>
<artifact><id>commons-lang:commons-lang:2.5</id></artifact>
<artifact><id>commons-lang:commons-lang:2.6</id></artifact>
<artifact><id>org.apache.commons:commons-lang3:3.1</id></artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
The fetch of transitive dependencies may be disabled using the <transitive>false</transitive> option on the <artifcat> level.
<artifact>
<id>commons-io:commons-io:2.1</id>
<transitive>false</transitive>
<instructions>
<Import-Package>*;resolution:=optional</Import-Package>
<Export-Package>*</Export-Package>
(…)
</instructions>
</artifact>
Finally, the default MANIFEST.MF can be regenerated and overwritten (specific Imports/Exports may be then specified) using the <override>true</override> option .
Then the generated site may be exposed using a jetty-plugin and consumed in the Eclipse IDE.
Please, see the examples section (I described everything there): https://github.com/reficio/p2-maven-plugin#examples
- How do you like that plugin? Do you think it is of any good?
- What should be improved?
http://dev.eclipse.org/mhonarc/lists/tycho-dev/msg00541.html
http://dev.eclipse.org/mhonarc/lists/tycho-user/msg02103.html
but because of some Maven/Plexus limitations it was not that simple. It's impossible now to invoke any Maven logic before Tycho, there are some problems with sharing classes across class-loaders.
My perfect case would be if the plugin was invoked seamlessly by Tycho during the compilation, so that the generated p2-site is automatically added to the Tycho repositories.
Then the perfect case would be if this repository could be referenced from Eclipse IDE, so that the project compiles in the IDE without any duplication in the dependency handling.
Right now, it's not that simple since the local-folder "repository" is not handled properly.
I would be very grateful for any comments.
Regards,
Tom Bujok
http://www.reficio.org
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder & CTO, Sonatype
Founder, Apache Maven
http://twitter.com/jvanzyl
---------------------------------------------------------
Simplex sigillum veri. (Simplicity is the seal of truth.)
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder & CTO, Sonatype
Founder, Apache Maven
http://twitter.com/jvanzyl
---------------------------------------------------------
Three people can keep a secret provided two of them are dead.
-- Benjamin Franklin
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
Jason van Zyl
2012-08-21 13:07:54 UTC
Permalink
Without creating something that referenceable from a target definition, no. But I believe you're duplicating much of what already exists.

In m2e for example it's already part of the target platform while developing so the wrapper bundle works great there (we install juno and install m2e and we have our target platform). The wrapper bundles have the distinct advantage of being able to run/debug projects you are working on from within the Eclipse workspace. The work we did creates a bridge between the Maven world and the Eclipse world through m2e (and the work done in PDE, the maven bundle plugin and m2e). I can work on Maven as a wrapper bundle with m2e seamlessly in the workspace. Our use case is a little different with m2e as working across the entire set of dependencies is important. Unfortunately Tycho/Maven just works entirely differently than PDE. I understand what you're trying to do where statically importing the Maven dependencies as part of the target definition is workable for you but seems not that much different than what's been done aside from making it referenceable from a target definition.

Maybe there is a way you can augment what already exists by creating something using the wrapper bundle method that would allow it to be referenced in a target platform definition?
Post by Tom Bujok
Hi,
- create a tycho RCP project
- declare a Maven dependency in pom.xml (with pomDependencies set to consider)
- import project to Eclipse IDE (with m2Eclipse)
Or am I wrong?
Tom
Tom Bujok
2012-09-04 13:53:10 UTC
Permalink
Hi,

Thanks for your answer and sorry for my late reply.

Somehow, I have the feeling that we are not comparing apples with apples in this discussion. I understand your point that you have to work across the entire set of dependencies (modify, compile, run/debug) so in your case the m2e wrapper bundler is the perfect match. It's not a common case, though, if you work on a custom RCP/Tycho project and you simply want to add a bunch of 3rd party dependencies such as commons, guava or mockito. You simply want to reference and use them in your project and then you want your project to compile in the console AND in the PDE. It's not a problem that they are statically included, since you hardly every want to modify them - and if so, it's no a problem to recompile and reinstall them to your local repo (or repository manager).

In order to provide you with more info about my solution (p2-maven-plugin) we could say that it does similar things that were provided by the nexus-p2-repository-plugin and the nexus-bundle-maker-plugin.
My stuff however:
- concentrates on 3rd party maven deps (not on eclipse plugins/features)
- supports source bundles (generates Eclipse-SourceBundles)
- fetches transitive dependencies (independently for every dependency -> not as a single dependency tree)
- enables the user to customize the bundler instructions (Import-Export packages)
- enables the user to overwrite the existing MANIFEST.MF (if required - not recommended though)
- is well documented -> http://github.com/reficio/p2-maven-plugin

As I understand you all see the need of such tool since you created the aforementioned nexus artifacts. I cannot understand why not to go one step further and not to include such a thing in the Tycho/PDE build cycle so that everybody is happy? I am not trying to duplicate anything - I have hardly written my own code in the p2-maven-plugin. It's an integration and compilation of the aether, bnd-tool, maven-bundle-plugin, tycho-p2-extras-plugin and CategoryPublisher tools. I am just trying to fill the gap that I find really annoying in the Eclipse / Tycho / Maven ecosystem. If somebody has a really corporate background it's probably not a problem to have an instance of nexus, artifactory or to configure advanced bash scripts to generate bundles and then build and deploy p2 sites. I am, however, trying to be very pragmatic, think in a simplistic way, and look at the problem as I knew nothing about RCP, OSGi or Tycho - I want the simplest possible solution just like I was using Maven in a standalone app.

So (to repeat myself), the problems that I find annoying in the current setup:
Problem nr. 1): I setup a Mavenized RCP project, import it to Eclipse, add an dependency to the pom.xml (that part I am acquainted with), and a bummer - it compiles in the console, but I cannot import it in the IDE. How this poor people could understand in 5 minutes that they need complex p2 proxies and nexus-pro editions to add a simple mockito.jar to their project in an automated and repeatable way (so they don't have to generate the MANIFEST.MF themselves and commit the jar into the lib folder)?
Problem nr. 2): I want to customize the generation of the MANIFEST.MF for the dependencies that are not OSGi bundles, in case when the artifact is not an OSGi bundle. How can I do it know with the tycho wrapper invoked behind the scenes?
Problem nr. 3): I want to add two versions of the same dependency (for example commons-lang 2.2 and 2.6) to the same project and import 2 packages from two different versions (OSGi enables me to do it), but the tycho-maven dependency resolution does not support it, since maven will ignore one of them.

This is really a complex stuff to configure, and unless you are really experiences in Maven, OSGi and Tycho you simply cannot solve it. RCP and Tycho are really popular in the DACH region (Germany, Switzerland, Austria) - and I have attended many JUG/conference sessions about Tycho - but every person presents a custom/complex/proprietary solution how to solve the problem of 3rd party dependencies. Every presentation contains a point which sounds more or less like this: "Problems with 3rd party dependencies". Here another example (in german though, however you will understand the keywords: Probleme, Third-Party): http://www.jugsaxony.org/?epl_event=bauen-mit-tycho

My goal was to make the usage of tycho and PDE as simple as the usage of Maven and IDE (in a standalone app). I just simply want to create a project, import it to eclipse, add a dependency (no matter if located in an update site, no matter if it's an OSGi bundle), compile and run it. It should be as simple as that.

So my proposal are as follows:
a.) external 3rd party dependencies are specified in the project descriptor (not as maven dependencies, to overcome the multiple version of the same artifact mismatch), generally speaking the pom-first tycho approach is not the best practice - so I would recommend using the manifest first approach
b.) Tycho build produces a referencable p2-update-sites somewhere in a local folder containing artifacts that are specified as 3rd party dependencies. The folder may be either in the project/target folder, or in the drive/.tycho folder (just like .m2).
c.) the wrap of 3rd party dependencies that are not bundles should be customizable (import-export packages, etc.)
d.) resolution of the 3rd party dependencies should include transitive dependencies (like in maven)
e.) Tycho dependency resolver should be extended so that it may consume p2-update sites that are located in a local folder
f.) m2e tycho plugin should be modified so that it automatically adds the 3rd party local update sites to the target definition (it may be manually included by the user as well, since PDE already supports update sites located in a local folder).

Using this approach I can also easily debug these dependencies (without modifying them of course) - I have just tested it since p2-maven-plugin now supports the generation of the Eclipse-SourceBundles.
All of these could be easily done with my p2-maven-plugin (or anything else, but that's not the point), to make the eclipse-rcp, tycho, maven, PDE environment more user-friendly for the technology newcomers…

I would really appreciate to get some constructive responses (positive/negative) from the community on that topic. So far I have not got it.
Cheers,
Tom

--
Tom Bujok
Reficio™ - Reestablish your software!
http://www.reficio.org
Post by Jason van Zyl
Without creating something that referenceable from a target definition, no. But I believe you're duplicating much of what already exists.
In m2e for example it's already part of the target platform while developing so the wrapper bundle works great there (we install juno and install m2e and we have our target platform). The wrapper bundles have the distinct advantage of being able to run/debug projects you are working on from within the Eclipse workspace. The work we did creates a bridge between the Maven world and the Eclipse world through m2e (and the work done in PDE, the maven bundle plugin and m2e). I can work on Maven as a wrapper bundle with m2e seamlessly in the workspace. Our use case is a little different with m2e as working across the entire set of dependencies is important. Unfortunately Tycho/Maven just works entirely differently than PDE. I understand what you're trying to do where statically importing the Maven dependencies as part of the target definition is workable for you but seems not that much different than what's been done aside from making it referenceable from a target definition.
Maybe there is a way you can augment what already exists by creating something using the wrapper bundle method that would allow it to be referenced in a target platform definition?
Post by Tom Bujok
Hi,
- create a tycho RCP project
- declare a Maven dependency in pom.xml (with pomDependencies set to consider)
- import project to Eclipse IDE (with m2Eclipse)
Or am I wrong?
Tom
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
Hugues Malphettes
2012-09-05 01:49:45 UTC
Permalink
Thanks Tom for sharing this to the community.

At Intalio we face the same issues; I used to be one of those
"dependency configurator".
We ended up rebuilding many of the bundles from their sources with
tycho just to get the MANIFEST.MF under control and to generate the
proper source bundles.

The m2e life-cycle does not fit what we need: we want to automate the
creation of p2 repositories from 3rd party jars available in maven
central.
Those jars most often need to have their manifests tweaked and never
come with a source bundle.

We wish this was supported natively in tycho: Jason if we are missing
something let us know.

We were looking at building our own plugin for this.
We are looking forward to try your plugin next time we work on this.

Best
Hugues
Post by Tom Bujok
Hi,
Thanks for your answer and sorry for my late reply.
Somehow, I have the feeling that we are not comparing apples with apples in
this discussion. I understand your point that you have to work across the
entire set of dependencies (modify, compile, run/debug) so in your case the
m2e wrapper bundler is the perfect match. It's not a common case, though, if
you work on a custom RCP/Tycho project and you simply want to add a bunch of
3rd party dependencies such as commons, guava or mockito. You simply want to
reference and use them in your project and then you want your project to
compile in the console AND in the PDE. It's not a problem that they are
statically included, since you hardly every want to modify them - and if so,
it's no a problem to recompile and reinstall them to your local repo (or
repository manager).
In order to provide you with more info about my solution (p2-maven-plugin)
we could say that it does similar things that were provided by the
nexus-p2-repository-plugin and the nexus-bundle-maker-plugin.
- concentrates on 3rd party maven deps (not on eclipse plugins/features)
- supports source bundles (generates Eclipse-SourceBundles)
- fetches transitive dependencies (independently for every dependency -> not
as a single dependency tree)
- enables the user to customize the bundler instructions (Import-Export packages)
- enables the user to overwrite the existing MANIFEST.MF (if required - not
recommended though)
- is well documented -> http://github.com/reficio/p2-maven-plugin
As I understand you all see the need of such tool since you created the
aforementioned nexus artifacts. I cannot understand why not to go one step
further and not to include such a thing in the Tycho/PDE build cycle so that
everybody is happy? I am not trying to duplicate anything - I have hardly
written my own code in the p2-maven-plugin. It's an integration and
compilation of the aether, bnd-tool, maven-bundle-plugin,
tycho-p2-extras-plugin and CategoryPublisher tools. I am just trying to fill
the gap that I find really annoying in the Eclipse / Tycho / Maven
ecosystem. If somebody has a really corporate background it's probably not a
problem to have an instance of nexus, artifactory or to configure advanced
bash scripts to generate bundles and then build and deploy p2 sites. I am,
however, trying to be very pragmatic, think in a simplistic way, and look at
the problem as I knew nothing about RCP, OSGi or Tycho - I want the simplest
possible solution just like I was using Maven in a standalone app.
Problem nr. 1): I setup a Mavenized RCP project, import it to Eclipse, add
an dependency to the pom.xml (that part I am acquainted with), and a bummer
- it compiles in the console, but I cannot import it in the IDE. How this
poor people could understand in 5 minutes that they need complex p2 proxies
and nexus-pro editions to add a simple mockito.jar to their project in an
automated and repeatable way (so they don't have to generate the MANIFEST.MF
themselves and commit the jar into the lib folder)?
Problem nr. 2): I want to customize the generation of the MANIFEST.MF for
the dependencies that are not OSGi bundles, in case when the artifact is not
an OSGi bundle. How can I do it know with the tycho wrapper invoked behind
the scenes?
Problem nr. 3): I want to add two versions of the same dependency (for
example commons-lang 2.2 and 2.6) to the same project and import 2 packages
from two different versions (OSGi enables me to do it), but the tycho-maven
dependency resolution does not support it, since maven will ignore one of
them.
This is really a complex stuff to configure, and unless you are really
experiences in Maven, OSGi and Tycho you simply cannot solve it. RCP and
Tycho are really popular in the DACH region (Germany, Switzerland, Austria)
- and I have attended many JUG/conference sessions about Tycho - but every
person presents a custom/complex/proprietary solution how to solve the
problem of 3rd party dependencies. Every presentation contains a point which
sounds more or less like this: "Problems with 3rd party dependencies". Here
Probleme, Third-Party): http://www.jugsaxony.org/?epl_event=bauen-mit-tycho
My goal was to make the usage of tycho and PDE as simple as the usage of
Maven and IDE (in a standalone app). I just simply want to create a project,
import it to eclipse, add a dependency (no matter if located in an update
site, no matter if it's an OSGi bundle), compile and run it. It should be as
simple as that.
a.) external 3rd party dependencies are specified in the project descriptor
(not as maven dependencies, to overcome the multiple version of the same
artifact mismatch), generally speaking the pom-first tycho approach is not
the best practice - so I would recommend using the manifest first approach
b.) Tycho build produces a referencable p2-update-sites somewhere in a local
folder containing artifacts that are specified as 3rd party dependencies.
The folder may be either in the project/target folder, or in the
drive/.tycho folder (just like .m2).
c.) the wrap of 3rd party dependencies that are not bundles should be
customizable (import-export packages, etc.)
d.) resolution of the 3rd party dependencies should include transitive
dependencies (like in maven)
e.) Tycho dependency resolver should be extended so that it may consume
p2-update sites that are located in a local folder
f.) m2e tycho plugin should be modified so that it automatically adds the
3rd party local update sites to the target definition (it may be manually
included by the user as well, since PDE already supports update sites
located in a local folder).
Using this approach I can also easily debug these dependencies (without
modifying them of course) - I have just tested it since p2-maven-plugin now
supports the generation of the Eclipse-SourceBundles.
All of these could be easily done with my p2-maven-plugin (or anything else,
but that's not the point), to make the eclipse-rcp, tycho, maven, PDE
environment more user-friendly for the technology newcomers…
I would really appreciate to get some constructive responses
(positive/negative) from the community on that topic. So far I have not got
it.
Cheers,
Tom
--
Tom Bujok
Reficio™ - Reestablish your software!
http://www.reficio.org
Without creating something that referenceable from a target definition, no.
But I believe you're duplicating much of what already exists.
In m2e for example it's already part of the target platform while developing
so the wrapper bundle works great there (we install juno and install m2e and
we have our target platform). The wrapper bundles have the distinct
advantage of being able to run/debug projects you are working on from within
the Eclipse workspace. The work we did creates a bridge between the Maven
world and the Eclipse world through m2e (and the work done in PDE, the maven
bundle plugin and m2e). I can work on Maven as a wrapper bundle with m2e
seamlessly in the workspace. Our use case is a little different with m2e as
working across the entire set of dependencies is important. Unfortunately
Tycho/Maven just works entirely differently than PDE. I understand what
you're trying to do where statically importing the Maven dependencies as
part of the target definition is workable for you but seems not that much
different than what's been done aside from making it referenceable from a
target definition.
Maybe there is a way you can augment what already exists by creating
something using the wrapper bundle method that would allow it to be
referenced in a target platform definition?
Hi,
- create a tycho RCP project
- declare a Maven dependency in pom.xml (with pomDependencies set to consider)
- import project to Eclipse IDE (with m2Eclipse)
Or am I wrong?
Tom
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
_______________________________________________
tycho-dev mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-dev
Loading...