Accessing Code of some major Open Source Projects
Cool, open source projects are of high quality if thousands test them. They should have some excellent build and test structure. Perhaps coding style as well? There should be plenty of good design patterns to pick up.
I love exploring and I explored open source projects 10 years ago as well. It should be just download, run setup and browse code, right? Let's see. How difficult can it be with the tools of today?
1. Eclipse 4.6 Neon - This is the environment where I will be accessing the projects, building and exploring them. The download was easy. When running it you'll need to select a directory as workspace where all your project files will land. With these big projects we will explore, there will be upwards of a hundred thousand files. This workspace directory should preferably be located on an SSD drive for faster access for many tiny files. When Eclipse is up, go to workspace and define git default repo directory via Windows --> Preferences --> Team --> Git to match your workspace directory. Now, for the extensions part. There are plenty of extras to download, so this will take some time. Go to Help --> Install New Software for the first install part. For the second install part, you can find additional extensions and popular ones in Help --> Marketplace. Need Java EE, Web, JavaScript, Git, Maven, etc. As I plan to run JBoss/Wildfly, I also get JBoss Tools, which has a lot of stuff inside to pick among as well. Once Eclipse is up and running, go to Git perspective and add Git repositories as linked below. Just copy the repository links one at a time and paste it (CTRL-V) inside Eclipse. In the branch selection, you can select the proper version. The last stable version is a wise choice. At one point during the work with the projects below, a m2e connector for build-helper-maven-plugin was added. Also, adding Groovy for Eclipse will avoid some compiler errors for Spring: Go to Help --> Install New Software and add org.codehaus.groovy.eclipse.site - http://dist.springsource.org/snapshot/GRECLIPSE/e4.6/ using the Add button in upper right corner. Select the required ones and Compiler 2.4 as well as JDT Core patch for Groovy-Eclipse plugin on Eclipse 4.6.
2. Maven 3 - very popular everything tool. I quickly found out this was needed as part of project installs, so I downloaded it and added it to PATH to be able to run mvn from command line. This is optional, as Maven can easily be run from within Eclipse by right-clicking project and select Run/Run as Maven..
3. Spring Framework 4.3 - held in high regard for helping to get a good design on server side stuff. The spring team have made a useful batch file that builds eclipse project files. Once spring git repo is added to Eclipse (paste this in eclipse git repo view), run CMD (CTRL-R, enter CMD), go to the Working Tree and run import-into-eclipse.bat. Follow the steps. It just works. The build is a success.
4. Hibernate-orm 5.1 - a mighty middleware for connecting the application data layer to a relational database. Aged only 15 this year, it is still much younger than the SQL on the other side of the DBMS. There are many hibernate repositories, for now, go with hibernate-orm git repo (paste this in eclipse git repo view) and select the 5.1 one (the last as of this moment). See the hibernate guide to contribution using eclipse. It says how to build eclipse project files. After importing the existing project files to Eclipse and setting Java 8 as compiler level, there are still some errors in a few projects. Time to turn creative and issue a .\gradlew build command to build and test the project. Hibernate has about 4500 tests it runs. 3 of these failed. However, .\gradlew jar builds the jars.
- After refreshing the project in Eclipse, some projects still has some problems. I suspect it is caused by some missing extension in Eclipse. For instance, some classes like Customer_ and Person_ is not found. The equivalent Customer and Person is present. Searching for 'hibernate Person_ cannot be resolved to a variable CriteriaTest.java /documentation/src/test/java/org/hibernate/userguide/criteria' gave no easily accessible solutions.
- Another error is some gradle cache files (below apache.karaf) was unable to read. I looked at them, to me they looked like valid XML. I deleted them, but another build did not rebuild them.
- Found out Hibernate 5.1.1 was announced in the blog in.relation.to. It seems 5.2.x smells better, described with Java 8, JCache support, hibernate-entitymanager consolidation. Resets to 5.2.2. Resetting is a simple function in git but takes a lot of time, if you process the whole tree as I told it to do. I suppose I needed to process the whole tree. This did not make out. With my limited git experience, I got confused as to what version of source I ended up with. So decided to delete all hibernate-project files and start afresh by pulling 5.2.2 from git repo. The problem is, there is no 5.2.x in git repo, you'll have to get the 5.1 branch and then somehow go sideways with git to get the 5.2.2. I have yet to discover how to do this with git. I tried 'reset' to a tag, but unsure if this is correct.
- Oh well, can't have it all I guess. At least the jar was built using gradlew. Proceeding to next project.
5. Jenkins 2.7.3 - a cool apparatus for continuous integration and building of projects. Judging by the number of plugins made for it, Jenkins is useful for many developers. From the jenkins git repository, get a stable branch, but which one? (paste this in eclipse git repo view). Follow beginners guide to contributing. Run mvn Plight-test install to produce additional source files. I found fails on branches 2.7 and 2.2.1 as well as stable-2.7.
- Continuing the Jenkins build, now run mvn -DdownloadSources=true eclipse:eclipse to produce the eclipse project files. This will surprisingly also install node and npm (locally to jenkins/war/node sub-directory). It also produces a 67 MB war file for test. Interestingly, it also downloads the sources to subversion. Better sooner than never, I imagine! After about 11 min, the build is a success.
6. Apache James - enables us with a programmatic interface to send and receive email. There is a nice intro to James 1 written in 2003.
Plan draft:
- look into how to set up James as MTA (mail transfer agent).
- look into how to create a MUA (mail user agent) using Maillets with some functions.
- identify where (in MTA or MUA) to put the functional requirements:
FA1 Add recipient (with its external email address)
FA2 Delete recipient
FB1 Create group
FB2 List recipients of a group
FB3 Add recipient to a group
FB4 Delete recipient from a group
FB5 Delete group
FC1 A message sent to a group is forwarded to all recipients of the group
Other requirements:
O1 A recipient can be in one or more groups
O2 A recipient is also known as subscriber, member
7. What's next?
- Hibernate Search (add mighty search capabilities to your site)
- Hibernate Validator (add validation in a smart way?)
- Apache Lucene (empowers the search capabilities beneath hibernate search)
- Wildfly (java app server + more)
- Apache httpd (web server)
- Apache Tomcat (java app server)
- Apache Jetspeed 2 (portal) - see building Jetspeed 2 with Maven 2 (from Jetspeed 2.3 with Maven 3)
I love exploring and I explored open source projects 10 years ago as well. It should be just download, run setup and browse code, right? Let's see. How difficult can it be with the tools of today?
1. Eclipse 4.6 Neon - This is the environment where I will be accessing the projects, building and exploring them. The download was easy. When running it you'll need to select a directory as workspace where all your project files will land. With these big projects we will explore, there will be upwards of a hundred thousand files. This workspace directory should preferably be located on an SSD drive for faster access for many tiny files. When Eclipse is up, go to workspace and define git default repo directory via Windows --> Preferences --> Team --> Git to match your workspace directory. Now, for the extensions part. There are plenty of extras to download, so this will take some time. Go to Help --> Install New Software for the first install part. For the second install part, you can find additional extensions and popular ones in Help --> Marketplace. Need Java EE, Web, JavaScript, Git, Maven, etc. As I plan to run JBoss/Wildfly, I also get JBoss Tools, which has a lot of stuff inside to pick among as well. Once Eclipse is up and running, go to Git perspective and add Git repositories as linked below. Just copy the repository links one at a time and paste it (CTRL-V) inside Eclipse. In the branch selection, you can select the proper version. The last stable version is a wise choice. At one point during the work with the projects below, a m2e connector for build-helper-maven-plugin was added. Also, adding Groovy for Eclipse will avoid some compiler errors for Spring: Go to Help --> Install New Software and add org.codehaus.groovy.eclipse.site - http://dist.springsource.org/snapshot/GRECLIPSE/e4.6/ using the Add button in upper right corner. Select the required ones and Compiler 2.4 as well as JDT Core patch for Groovy-Eclipse plugin on Eclipse 4.6.
2. Maven 3 - very popular everything tool. I quickly found out this was needed as part of project installs, so I downloaded it and added it to PATH to be able to run mvn from command line. This is optional, as Maven can easily be run from within Eclipse by right-clicking project and select Run/Run as Maven..
3. Spring Framework 4.3 - held in high regard for helping to get a good design on server side stuff. The spring team have made a useful batch file that builds eclipse project files. Once spring git repo is added to Eclipse (paste this in eclipse git repo view), run CMD (CTRL-R, enter CMD), go to the Working Tree and run import-into-eclipse.bat. Follow the steps. It just works. The build is a success.
4. Hibernate-orm 5.1 - a mighty middleware for connecting the application data layer to a relational database. Aged only 15 this year, it is still much younger than the SQL on the other side of the DBMS. There are many hibernate repositories, for now, go with hibernate-orm git repo (paste this in eclipse git repo view) and select the 5.1 one (the last as of this moment). See the hibernate guide to contribution using eclipse. It says how to build eclipse project files. After importing the existing project files to Eclipse and setting Java 8 as compiler level, there are still some errors in a few projects. Time to turn creative and issue a .\gradlew build command to build and test the project. Hibernate has about 4500 tests it runs. 3 of these failed. However, .\gradlew jar builds the jars.
- After refreshing the project in Eclipse, some projects still has some problems. I suspect it is caused by some missing extension in Eclipse. For instance, some classes like Customer_ and Person_ is not found. The equivalent Customer and Person is present. Searching for 'hibernate Person_ cannot be resolved to a variable CriteriaTest.java /documentation/src/test/java/org/hibernate/userguide/criteria' gave no easily accessible solutions.
- Another error is some gradle cache files (below apache.karaf) was unable to read. I looked at them, to me they looked like valid XML. I deleted them, but another build did not rebuild them.
- Found out Hibernate 5.1.1 was announced in the blog in.relation.to. It seems 5.2.x smells better, described with Java 8, JCache support, hibernate-entitymanager consolidation. Resets to 5.2.2. Resetting is a simple function in git but takes a lot of time, if you process the whole tree as I told it to do. I suppose I needed to process the whole tree. This did not make out. With my limited git experience, I got confused as to what version of source I ended up with. So decided to delete all hibernate-project files and start afresh by pulling 5.2.2 from git repo. The problem is, there is no 5.2.x in git repo, you'll have to get the 5.1 branch and then somehow go sideways with git to get the 5.2.2. I have yet to discover how to do this with git. I tried 'reset' to a tag, but unsure if this is correct.
- Oh well, can't have it all I guess. At least the jar was built using gradlew. Proceeding to next project.
- Continuing the Jenkins build, now run mvn -DdownloadSources=true eclipse:eclipse to produce the eclipse project files. This will surprisingly also install node and npm (locally to jenkins/war/node sub-directory). It also produces a 67 MB war file for test. Interestingly, it also downloads the sources to subversion. Better sooner than never, I imagine! After about 11 min, the build is a success.
6. Apache James - enables us with a programmatic interface to send and receive email. There is a nice intro to James 1 written in 2003.
Plan draft:
- look into how to set up James as MTA (mail transfer agent).
- look into how to create a MUA (mail user agent) using Maillets with some functions.
- identify where (in MTA or MUA) to put the functional requirements:
FA1 Add recipient (with its external email address)
FA2 Delete recipient
FB1 Create group
FB2 List recipients of a group
FB3 Add recipient to a group
FB4 Delete recipient from a group
FB5 Delete group
FC1 A message sent to a group is forwarded to all recipients of the group
Other requirements:
O1 A recipient can be in one or more groups
O2 A recipient is also known as subscriber, member
- Hibernate Search (add mighty search capabilities to your site)
- Hibernate Validator (add validation in a smart way?)
- Apache Lucene (empowers the search capabilities beneath hibernate search)
- Wildfly (java app server + more)
- Apache httpd (web server)
- Apache Tomcat (java app server)
- Apache Jetspeed 2 (portal) - see building Jetspeed 2 with Maven 2 (from Jetspeed 2.3 with Maven 3)