Q: Your web site states that the "top scoring teams will be invited to give presentations at the special session, and submit papers to IJPHM". Does this means that only the first two (one from each category) will be invited?
A: No. We expect several competitors from both categories will be invited to present, depending on how they do.
Monday, May 25, 2009
FAQ#8: releasing the answers
Q: Are you going to post the solutions of the Data Challenge, e.g., after the competition?
A: Probably not all of them.
As with last year, we aren't releasing the full data set. Instead, we are holding on to it for use as a "blind standard" for comparing algorithms.
A: Probably not all of them.
As with last year, we aren't releasing the full data set. Instead, we are holding on to it for use as a "blind standard" for comparing algorithms.
Thursday, May 21, 2009
FAQ#7: Experimental Conditions II
Q: We would like to ask you to clarify your comment in FAQ#6. We are confused and not sure that we understood it correctly.
What do you mean by "a case"? Did you mean that "a case" is a specific combination of good and bad components, and that over all there are only 14 such combinations?
Are you saying that the same combination of faults is replicated 40 times (4 times in each combination of speed and load)?
If the answer to the last question is yes, the implication would be 40 identical lines in the results file for each of the 14 cases. Is that correct?
A: Yes, the same combination of faults is replicated 40 times.
However, the goal of the competition is to promote the exchange of innovative ideas and to advance PHM as a scientific discipline, and, this year in particular, to fault detection and magnitude estimation for a generic gearbox using accelerometer data and information about bearing geometry. Participants are scored based on their ability to correctly identify type, location, and magnitude and damage in a gear system.
Algorithms that rely on knowledge of this specific data set will be disqualified.
What do you mean by "a case"? Did you mean that "a case" is a specific combination of good and bad components, and that over all there are only 14 such combinations?
Are you saying that the same combination of faults is replicated 40 times (4 times in each combination of speed and load)?
If the answer to the last question is yes, the implication would be 40 identical lines in the results file for each of the 14 cases. Is that correct?
A: Yes, the same combination of faults is replicated 40 times.
However, the goal of the competition is to promote the exchange of innovative ideas and to advance PHM as a scientific discipline, and, this year in particular, to fault detection and magnitude estimation for a generic gearbox using accelerometer data and information about bearing geometry. Participants are scored based on their ability to correctly identify type, location, and magnitude and damage in a gear system.
Algorithms that rely on knowledge of this specific data set will be disqualified.
Monday, May 11, 2009
FAQ#6: Experimental Conditions
Some explicit guidance on the experimental conditions.
There are six helical cases and eight spur cases; within each case, there are four replications (repeated test conditions), five speeds, and two loads.
So: (6*4*5*2)+(8*4*5*2) = 560 files
There are six helical cases and eight spur cases; within each case, there are four replications (repeated test conditions), five speeds, and two loads.
So: (6*4*5*2)+(8*4*5*2) = 560 files
Friday, May 1, 2009
FAQ#5: Submitting results & scoring
Q: How often can we submit results, how are they scored, and how frequently is the leaderboard updated?
A: You can submit results once every 24 hours. The leaderboard is updated once every day, at approximately noon PST.
Results are scored by calculating the Hamming distance between the submitted results file, and the ground truth.For example, if the true state of the system is [1,0,0,1,0] and you submit [1,1,0,0,0,0], your score is 2.
A: You can submit results once every 24 hours. The leaderboard is updated once every day, at approximately noon PST.
Results are scored by calculating the Hamming distance between the submitted results file, and the ground truth.For example, if the true state of the system is [1,0,0,1,0] and you submit [1,1,0,0,0,0], your score is 2.
FAQ#4: Hey, people might cheat!
Q: Will the leaderboard allow the participants use the evaluation results as "feedback" to tune their algorithms?
A: Please do not do this! It is pointless...
The Data Challenge is about promoting the exchange of innovative ideas and advancing PHM as a scientific discipline. Please compete fairly, and do not try to game the system. You may score well on the leaderboard, but ultimately you will have to publish your approach, and if it is incapable of achieving similar results on a holdout dataset, you will be disqualified.
Note that in the spirit of fair competition, we allow only one account per team. Please do not register multiple times under different user names, under fictitious names, or using anonymous accounts. Competition organizers reserve the right to delete multiple entries from the same person (or team) and will aggressively disqualify those who are trying to game the system or use fictitious identities.
A: Please do not do this! It is pointless...
The Data Challenge is about promoting the exchange of innovative ideas and advancing PHM as a scientific discipline. Please compete fairly, and do not try to game the system. You may score well on the leaderboard, but ultimately you will have to publish your approach, and if it is incapable of achieving similar results on a holdout dataset, you will be disqualified.
Note that in the spirit of fair competition, we allow only one account per team. Please do not register multiple times under different user names, under fictitious names, or using anonymous accounts. Competition organizers reserve the right to delete multiple entries from the same person (or team) and will aggressively disqualify those who are trying to game the system or use fictitious identities.
FAQ#3: J'accuse!
Q: The PHM challenge should also allow researchers from the model-free camp to participate. This means that pattern recognition based supervised learning techniques should be able to be used. Since the data is not labeled, it is not possible to run experiments related to classification performance. The authors of the challenge should provide two different data sets, one labeled for the learning and one unlabelled for the testing. Currently only model-based fault diagnosis can be done. This excludes a complete branch of research.
A: Any format will favor some group of researchers. Last year, the competition was exactly what you asked for: essentially a homework problem, with neatly packaged labeled data that gave machine learning researchers a huge advantage. Next year, some other group will assuredly have an advantage.
However, we have made every effort to level the playing field. We have provided background on domain fundamentals; Matlab code for algorithms to extract features from the data; and links to excellent papers on the analysis of this type of data. Moreover, we believe that the problem is difficult enough that very innovative approaches may be required to solve it: researchers who are looking at the problem from a "fresh perspective" may actually have an advantage...
A: Any format will favor some group of researchers. Last year, the competition was exactly what you asked for: essentially a homework problem, with neatly packaged labeled data that gave machine learning researchers a huge advantage. Next year, some other group will assuredly have an advantage.
However, we have made every effort to level the playing field. We have provided background on domain fundamentals; Matlab code for algorithms to extract features from the data; and links to excellent papers on the analysis of this type of data. Moreover, we believe that the problem is difficult enough that very innovative approaches may be required to solve it: researchers who are looking at the problem from a "fresh perspective" may actually have an advantage...
FAQ#2: labeled data
Q: Will some normal operational data or faulty data with ground truth be provided to help calibrate the algorithms?
A: No. The truth is in the data and the system geometry information.
A: No. The truth is in the data and the system geometry information.
More help getting started...
This paper is an excellent introduction to gearbox data analysis.
This paper is a good introduction to time synchronous averaging.
This paper is a good introduction to time synchronous averaging.
Subscribe to:
Posts (Atom)